Unit Testing Comparison Videos

In my presentation "Unit Testing Makes Me Faster", I show a couple of videos that compare manual testing and unit testing. Folks have asked me to make these videos available, but they don't really make sense without context. So, we're going to look at both here.

The Method
Here's the method that we have in our code that needs to be tested:

The particulars aren't important. This method checks to see if a credit card number passes the Luhn algorithm -- basically a checksum on a credit card number. We need to make sure that this method works.

If you want to look at the code for this, you can get it here: Unit Testing Makes Me Faster.

Manual Testing
Before I started unit testing, I would commonly throw together a tester application just to sanity check the algorithm. This would be a simple application with some input/output fields and a button.

In this case, I built a little WPF application (since WPF is my UI of choice these days). This application just takes a few minutes to build, and we end up with something that looks like this:

The code-behind for the button calls our method and puts the results in an output block:

To use this application, we paste our test numbers into the text box, click the button, and then check the output. This will tell us whether the "PassesLuhnCheck" method returns true or false:

Unit Testing
Building unit tests for this method is pretty simple. We just create a method to check for "true" results and a method to check for "false" results. To make things easier, we can parameterize the methods so that we can check multiple numbers.

Here's our check for the numbers that should pass the Luhn check (and thus return "true"):

Because we're using parameterized tests, we can check 10 different cases with this one method.

For more information on parameterized tests, see Parameterized Tests with NUnit.

We can do the same for the numbers that should not pass the Luhn check:

This gives us 5 more test cases that should all return "false".

Build Comparison
And this takes us to the comparison videos. In the first video, we show side-by-side building the tester application and implementing the unit tests. This starts from scratch (File -> New Project) for both options.

So that we don't have to wait too long, this video is sped up 3 times faster than normal:


Direct video link: Side-By-Side Test Build

What this shows is that when we're comfortable with our tools, it takes about the same amount of time to build either test solution -- about 5 minutes in this case.

There is a big caveat there: when we're comfortable with our tools.

In my case, I am comfortable with WPF XAML and with unit testing using NUnit. If I did not know WPF, then I would expect that building the tester application would take me a bit longer. In the same way, if I were new to unit testing and NUnit, I would expect that building the unit tests would take me a bit longer.

But if I'm comfortable in both environments, then we get a fair comparison between these solutions.

The build times may be similar, but we see a real difference when we look at regression testing.

Regression Comparison
Regression testing is when we make sure that we didn't break any existing functionality in the code. It's a bit hard to tell in the above video, but we do have some failing tests. If the input contains a non-numeric (such as a letter or a minus sign), then the method we're testing throws an exception.

Our job is to fix the method so that our tests will pass (without throwing exceptions), and then we need to go back and check that we didn't break the existing functionality.

This video shows a side-by-side comparison of fixing the problem and then running regression tests. This video is *not* sped up; it is real time:


Direct video link: Side-By-Side Regression Test

This is where we see a huge difference. With our unit tests in place, after we make the change to our code, we simply build and re-run the tests. The tests take 3 seconds to run. We get confirmation that our changes fixed the problem and also that we didn't break the existing functionality.

For the manual test application, we need to go through our test cases and copy/paste them into the application and click the button. This only takes about a minute to do. But even so, it's painful to watch. If you watch closely, you'll see that the unit tests are complete before we have a chance to check the first value manually.

The real question is which type of regression are we most likely to do? If the tests run automatically in 3 seconds, then I'm going to be running those tests all day long. If I have to go through a manual process of copy/paste and clicking, then I'm never going to do that.

Wrap Up
Doing side-by-side comparisons of various testing methods can give us a good idea of where the advantages lie. Granted, it takes time to get up-to-speed on unit testing, but that's no different from any other framework. And there are many other advantages to having automated tests in place as well. Check out the materials for some more information: Unit Testing Makes Me Faster.

Happy Coding!