Teaching a CNN to read handwriting

The project had four aims, each depending on the one before it.

1. Build a convolutional model that could be trained on a database of images.
2. Process and prepare a database of handwritten-character images to train it on.
3. Train it effectively enough to recognise characters with high accuracy.
4. Point the finished model at a single image file and have it read the character.

Almost all of the work went into the data. I trained on the EMNIST ByClass dataset: over 700,000 images spanning 62 classes of handwritten characters, digits plus upper- and lower-case letters, each one paired with its label.

Every image is a 28×28 grid of single-channel pixels, representing a transposed image of a letter. These matrices were small enough to train on quickly, but detailed enough to tell an a from a q.

The model receives a field of brightness values, one channel deep: 784 numbers per character.

Real handwriting is not perfectly centred or upright, so I augmented the data, randomly rotating and translating the training images, so that the model learned the shape of a character rather than one exact rendering of it.

To keep it from overfitting, I used learning-rate scheduling, easing the step size down as training progressed, together with early stopping, halting before the model started fitting noise in the training set instead of the signal. Together they prevented overfitting, so accuracy held on completely unseen data.

The model reached an average recognition rate of 82% on unseen, augmented data, and recognised characters I drew and fed in myself. The original aim was to point it at an image file and have it read the character, which it does.

About the missesMost of what it gets wrong are pairs that are ambiguous without context: K vs k, 0 vs O, 1 vs l. On some of those, such as C vs c, the model distinguished them more reliably than I did.

I am now building an end-to-end encrypted messaging protocol and desktop client for my A-level coursework, intended to replace SMTP and IMAP with encryption by default and no server access to plaintext or keys. Implementation in Rust begins in September.