Parameterized Tests with NUnit

In the last article, we implemented the rules for Conway's Game of Life using Test-Driven Development. For our testing framework, we used MSTest, but we did run into a limitation -- we had to create separate test methods for each test case even when the only difference was an input value.

I like MSTest because it's easy to get started with: (1) it's there -- nothing else to install, (2) creating a test project is as easy as "File | New Project", (3) the test runner is integrated with Visual Studio, and (4) it has the "Run Tests After Build" button (in some versions of Visual Studio). This automatically re-runs affected tests when you rebuild a project.

But it does lack parameterized tests (unless you're building Windows Store Apps). So, let's use a different testing framework that *does* support this: NUnit.

Completed code can be downloaded here: http://www.jeremybytes.com/Downloads.aspx#ConwayTDD
[Editor's Note: I fixed the bad link here - it was pointing to "localhost". D'oh]

Installing NUnit
I'm going to give the "NUnit noob" instructions for getting started since I was a "noob" not long ago. A big barrier to using a different framework is "where do I start?" Fortunately, we can get things through NuGet, and it integrates fairly well with our projects.

NuGetting NUnit
We'll start by adding a new project to our solution. This will be a class library project called "Conway.Library.NUnit". We'll call the initial class "LifeRulesTests" (this is the same as we used for the MSTest project):

Now, we just need to add NUnit. For this, I'll right-click on the new project and select "Manage NuGet Packages". Then we'll search online for NUnit:

And we'll install the first 2 items that come up: NUnit (the testing framework) and NUnit.Runners (which will give us the test runner).

When we install NUnit for the current project (Conway.Library.NUnit), it automatically adds a project reference to "nunint.framework" as well as a "packages.config" that makes sure we have the right package available to the project.

The Test Runner
When we install NUnit.Runners, it creates a solution-level package configuration. It also puts the test runner into our solution folder. If we right-click on the solution and choose "Open Folder in File Explorer", we can navigate down to the "tools" folder:

That's where we'll find "nunit.exe". Run this, and we'll end up with the NUnit test runner GUI:

We'll come back to this in just a bit.

Now, you're probably thinking that having the test runner in the solution folder is not a good idea. After all, if we want to use NUnit with multiple solutions, we would end up with lots of copies of this test runner.

And you're absolutely right. For long-term usage, I would move the test runner folder to a common location that can be easily shared. But we won't worry about that for now.

Building Our First Test
To make sure that everything is working, we'll just bring over one of our existing tests. We'll use the second test that we wrote for MSTest.

The syntax is a little bit different. Notice that our class is decorated with the "TestFixture" attribute (instead of "TestClass"), and our method is decorated with "TestCase" (instead of  "TestMethod").

Let's build our new project and run the test.

Running the Test
To run the test, we just need to open the assembly for our test project in the NUnit runner. For this, we'll just navigate to the bin/Debug folder of the "Conway.Library.NUnit" project and open the "Conway.Library.NUnit.dll" file:

We've got quite a big tree structure here. The top of the tree is navigating through the "dots" of our namespace: Conway.Library.NUnit. Then we have the name of the test class: LifeRulesTests. Then we have the tests themselves.

Now this looks strange because "LiveCells_TwoOrThreeLiveNeighbors2_Lives" is there twice. This will make sense once we add some parameters.

When we run the test, it passes.

Now that we have things set up, we can start to look at parameterization.

Parameterizing Tests
Here's the problem with our current test: even though the test case says "TwoOrThreeLiveNeighbors", we are only testing it with the value of "2". But instead of adding another test for the "3" case, we can simply parameterize this test.

And parameterization is just what it sounds like: we add a parameter to our test method. Here's our parameterized test with some test cases configured:

Let's walk through this. First, notice that we have a parameter for our test: "liveNeighbors". Since we have this parameter, we removed the line from the code that created and initialized the variable of the same name. This tells us that the number of live neighbors will be passed in to the test.

In order to pass in these values, we add attributes for the different test cases. Notice that our "TestCase" attribute now has a parameter. And we have 2 attributes so that we can cover each of our values.

Finally, I removed the "2" from the test name since we are using this same method for both test cases.

Running the Parameterized Test
Now let's build, go back to the test runner, and re-run the test:

The tree makes a bit more sense now. We have a single test method with 2 different test cases. We can see that this one test was run with both "2" and "3" as parameters.

Additional Parameterized Tests
We can do this for our other tests as well. Here's the "over-crowding" test:

This one method works for all 5 test cases.

If we continue this, we end up with 6 test methods total (down from 18 originally):

This shows the tree collapsed to our 6 methods. But notice that the results say "Passed: 18". And we can expand our tree to see all of the test cases:

This shows all 18 test cases even though we only have 6 unique test methods.

Why So Many Tests?
This is way better than writing 18 tests. But does 6 still seem like a lot? It seems like we could combine the tests a bit more.

For example, if we look at the "less than 2" and "greater than 3" tests on the living cell, we see that the starting state is the same ("Alive") and the expected result state is the same ("Dead"). So maybe we could create a "LiveCell_LessThanTwoOrMoreThanThreeLiveNeighbors_Dies" test. This would have 7 test cases for a single test method.

Or we could get even more creative by adding more parameters. What if we passed in the initial state, the number of live neighbors, and the expected result state? Then we could technically have a single test method that could work for all 18 test cases.

And this is where we need to think about finding the right balance. We can't only think about "how can I write the fewest tests possible". We also need to think about how we react when a test fails. If we have clearly named tests, then when it fails, we have a good idea of what part of the code we need to fix. The tests are not simply to help us write the code, they are also to help us fix things when we break the code or when we need to make enhancements to the system.

My Reasoning
The reason that I settled on 6 tests is that the test methods correspond to the rules of our system.

A reminder of the rules:

1. Any live cell with fewer than two live neighbours dies, as if caused by under-population.
2. Any live cell with two or three live neighbours lives on to the next generation.
3. Any live cell with more than three live neighbours dies, as if by overcrowding.
4. Any dead cell with exactly three live neighbours becomes a live cell, as if by reproduction.

For our first 3 rules (acting on living cells), we have 3 corresponding test methods. For the last rule, we do have a test for a dead cell coming back to life, but we also need a test for the negative of this case (where a cell does not come back to life).

I might reduce things down to 5 tests by combining the dead cell tests. This would be "exactly 3 live neighbors comes to life" and "not equal to 3 live neighbors stays dead". But since we have the "greater than" and "less than" values on the live cell tests, it makes sense to me to have similar methods / naming for the dead cell tests.

Wrap Up
Parameterized tests are pretty awesome. It allows us to write fewer tests when we have multiple inputs that result in the same expected output. And it makes it easier for us to test different inputs. In the case of the rules for Conway's Game of Life, this is a natural fit.

We saw that NUnit allows us to easily set up parameters for our test methods and then use attributes to set up the various test cases for each test. And NUnit is not difficult to include in our project -- we can use NuGet to add the framework to our project and also bring down the test runner (if we don't already have it).

NUnit also has a lot more features. And the test runner has plenty of options, too. So, take a bit of time to explore what else is available.

But just thinking about parameterized tests should give us some ideas on how we can make our tests easier to write and navigate without an overwhelming amount of copy/paste between methods.

Happy Coding!