To take a look at this, we'll first see why we might want to deal with exceptions in our tests. Then we'll look at the way we would handle this with MSTest (and see a few shortcomings). Then we'll see how NUnit makes things much easier.
To see related articles, including the method creation using TDD, MSTest, and NUnit, check out Coding Practice with Conway's Game of Life.
Needing to Check for Exceptions
We used TDD to create the method to process the rules for Conway's Game of Life. Here's the method that we ended up with:
This method passes all of our tests, and it works fine in the application. But when we ran it through Smart Unit Tests (from the Visual Studio 2015 Preview), we found a problem. Here are the results of that test run:
This shows the parameters that were passed in to our method to test it. We want to pay attention to the "currentState" parameter. This is an enum called "CellState" which has 2 values: Alive and Dead.
But we can see that in addition to these values, Smart Unit Tests also created a test with a value of "2". This seems strange at first, but we need to remember that enums are simply integers underneath. That means that "2" is a valid value (meaning it will compile), but it's technically out of range.
Adding the Guard Clauses
To fix this problem, we added some guard clauses (full article). The guard clauses do some range checking on our parameters, and throw exceptions if we run into an issue:
This makes sure that our CellState enum is within the range of our valid values (using the "IsDefined" function), and we also check the "liveNeighbors" parameter since we expect the values will be between 0 and 8.
If we hit one of these guard clauses, we expect that an ArgumentOutOfRangeException will be thrown. In order to have good coverage in our unit tests, we need to test that functionality.
Testing for Exceptions with MSTest
We'll create a couple of tests using MSTest first. This will show us the problems that we run into when we try to test for exceptions.
In MSTest, we have an attribute that we can use: "ExpectedException". When we put this attribute on a test, then the test will only pass if the code throws that exception. If the code does not throw that exception, then we get a failing test.
Here are a couple of tests for our guard clauses:
We can see that both of these tests expect to come across an ArgumentOutOfRangeException: one for the "currentState" (our first parameter), and one for the "liveNeighbors" (our second parameter)
The Problem
But we have a problem with this code. Even though we're able to verify that an ArgumentOutOfRangeException was thrown, we aren't able to verify *which* ArgumentOutOfRangeException was thrown. If our code is incorrect, it could be that both of these throw exceptions based on the first guard clause (which would be a problem).
So we really need to be able to get more details out of the exception in order to adequately test this scenario.
Manual Workaround
It would be nice if there was an automatic way of doing this, but unfortunately, the ExpectedException attribute that we're using is too limited for what we need.
One solution is to skip the attribute and code things up manually. Here's what that code looks like:
Instead of using the attribute, we wrap our code in a try/catch block. This gives us access to the actual exception being thrown. So in our catch block, we can verify that it is the correct one. In this case, we compare the message to what we expect.
Notice the "Assert.Fail()" call that we have in the try block. This is very important when we do a test like this. If a test does not have any failing asserts (and does not throw any exceptions), then by default, it is a passing test.
Since we expect the "GetNewState" call will result in an exception, we need to explicitly fail the test if the exception is not thrown. In this test, we do not expect that we will ever hit the "Assert.Fail()" line of code (since we expect the exception). But if we do hit that line of code for some reason, our test will fail, and we'll know that something went wrong.
This pattern works, and I've used it in my own unit tests. But it is a bit tedious to write. It would be much nicer if we could simplify our tests when we're checking for exceptions.
It turns out that NUnit allows us to simplify our code, making the tests easier to write and maintain.
Testing for Exceptions with NUnit
NUnit has an extensive library to make testing easier. We'll look at 2 methods today: Throws and Catch. For more information, you can check out the documentation for Exception Asserts.
As a side note, NUnit does also have an attribute-based way of checking exceptions which is more extensive than MSTest, but the Exception Asserts is the preferred way of doing things today.
Let's walk through this test:
First, we set up our variables that we use for our parameters (and we can see that our CellState parameter is out of range).
Next, we use the "Assert.Throws" method. This takes a generic parameter for the type of exception that we're expecting. One thing that is interesting about this method is that it returns that exception (which we save in the "ex" variable). This means that we can inspect the exception later.
The "Assert.Throws" method has a parameter that is a "TestDelegate" type. This is a delegate that takes no parameters. And since we know how to use lambda expressions as delegates, we can just put our code right in here.
In our case, we simply call the "GetNewState" method with our test parameters. This is the block of code that we expect to throw an exception.
Finally, we use an Assert like we did in MSTest to check to see if this is the particular ArgumentOutOfRangeException that we expected here.
Advantages
There are a few advantages here. First, we can get a reference to the exception that was thrown (through the return value). This gives us something that we can inspect further.
Second, we limit our check for an exception to a particular line of code (in this case the call to "GetNewState"). If we wanted, we could use a multi-statement lambda expression to wrap more code. We can think of this code as the code that we would wrap in the "try" block in our manual check.
This means that if an exception is thrown somewhere else in the test method (even if it is an ArgumentOutOfRangeException), it will result in a test failure. We will only get a passing test if the exception is thrown in the block of code that is part of the "Assert.Throws" method call.
Best of all, we don't need to manually fail the test with "Assert.Fail". This is one less piece of code to worry about. Overall, our readability is better, and that means we'll be more likely to write and maintain these types of tests.
"Throws" vs. "Catch"
NUnit has another exception assert method called "Catch". "Throws" looks for a specific exception type (in our case "ArgumentOutOfRangeException").
"Catch" will look for a specific exception type *or* any of its descendants. Let's look at another example:
Notice that "Assert.Catch" has a generic type parameter of "ArgumentException". This means that this will work on any code that throws an ArgumentException or any exception that descends from ArgumentException -- including ArgumentOutOfRangeException and ArgumentNullException.
Since our code throws an ArgumentOutOfRangeException, this test still passes.
Prefer "Throws" Over "Catch"
As general advice, I would recommend preferring "Throws" over "Catch". The reason for this is that we should always be looking for as specific of an exception as possible. This is true when we write our exception handling in our code, and it should also be true when we're writing tests to verify exceptions.
Of course there are cases where we may want to use "Catch", but we should consider the implications before doing so.
Other Helper Classes
NUnit has a number of helper classes to make assertions easier. Check out the category list on the sidebar of the Assertions documentation.
I still need to dig into these classes, but they look very useful. As an example, let's look at the StringAssert class. This gives us a chance to rewrite the final assertion in our test:
So instead of using "Assert.IsTrue" and then having the parse the contents of the parameter, we can use the "StringAssert.Contains" method. This is a bit more readable since we know right away that we're looking to see if a string contains a particular value.
And the method parameters follow the standard "expected/actual" pattern that we have in other assertions. So we're saying that we expect to find "Invalid Cell State" somewhere in our actual exception message string.
Wrap Up
So we've seen that NUnit has some classes and methods that make it much easier to check for exceptions in our unit tests. And since we can plug NUnit into our Visual Studio Test Explorer, we can get the result of those tests right in our IDE with everything else.
This is just one way that NUnit can make our tests easier to read and maintain. There are many other ways as well. I'm just getting started on this journey, and I'll be sure to keep you up-to-date on other interesting and useful things that I find.
Happy Coding!
No comments:
Post a Comment