Last time, I added the ability to flag which items are incorrectly identified:
This time, I added code so that we could look at 2 different algorithms side-by-side:
I won't go into the UI code, because that's not the interesting bit here. The interesting bit (at least to me) is how the results differ.
As a reminder, after the application finishes populating the boxes, I had to go through manually and click the incorrect items to flag them. I'd love to be able to do this automatically, but that's kind of the point here -- the computer can't figure out what's right and wrong.
If you'd like to look at the code, head to GitHub and look at the jeremybytes/digit-display project. The code here is from the "Comparison" branch.
Prediction Comparison
The way that this works is that each classifier is trained with the same training set (40,000 bitmaps). Then each classifier will compare each bitmap to the training set to find the closest match.
The idea of what is "closest" is determined by the specific algorithm. I won't go into the details here (you can read Mathias' book for the simplicity and complexity of this -- Machine Learning Projects for .NET Developers (Amazon link).
The first set uses the Manhattan Distance (Wikipedia link for the curious). The second set uses the slightly more complex Euclidean Distance (Wikipedia link).
What I was really curious about when I was first reading about the success of each at predicting hand-written digits, was if one simply made fewer errors (but mostly the same errors) or if one makde completely different errors from the other.
This initial visualization gives the answer to that question.
Each classifier was asked to predict values for the same 300 hand-written digits. You'll notice that the digits are in slightly different orders in each section. That's because the values are being processed in parallel, and the exact order will vary depending on the task scheduler.
Overall
Manhattan Distance: 9 errors out of 300 (97.0% accuracy)
Euclidean Distance: 7 errors out of 300 (97.7% accuracy)
Don't put too much into these results. We're working with a fairly small data set (300 items). We'll need to look at more / different data to come to firm conclusions.
Mostly the Same Errors
In scanning through the results we can see that they had trouble with the same items (click on the image for a larger version):
There were a couple of items where the Euclidean distance was able to successfully identify items that the Manhattan distance got incorrect, in this case an "8":
And here's another example -- also an "8":
This shows an improved accuracy (at least with this training set and this particular data set).
Different Answers
One interesting value was incorrectly identified by both classifiers; however, they came up with different answers:
In both cases, this "7" was incorrectly identified -- and that's understandable; this is a really bad "7". This is what I want to explore further.
Next Steps
I mentioned earlier that I want to add some more features to this to make it easier to explore the results.
This includes adding some inputs to control the size of the training set. A smaller training set means that the predictions will run more quickly, but they will also be less accurate.
In addition, I'd like to add an offset into the data. Right now, we're starting at the beginning of the data set and taking the first 300 values (in this case). I'd like to be able to start at an arbitrary location in the data set so that we can compare different parts of the data side-by-side.
I'll keep adding to this project. The UI code is getting a bit unwieldy already, so there will be some refactoring and clean up along the way as well.
I'm not sure where all of this is headed. But it's interesting for me to see these similarities and differences. And it's interesting to think about other possibilities. Recognizing hand-written digits is really easy for a human and not easy for a machine. This is getting me to think some more about what it really means to be human.
Happy Coding!
No comments:
Post a Comment