More Smart Unit Tests (Preview): Guard Clauses

Yesterday we looked at the new Smart Unit Tests feature that is part of Visual Studio 2015 Preview. I couldn't resist doing a few more updates to my code to see how Smart Unit Tests would react. As a reminder, this is based on preview versions of the technology (from 11/12/2014).

[You can get links to all of the articles in the series on my website.]

[Update: Smart Unit Tests has been released as IntelliTest with Visual Studio 2015 Enterprise. My experience with IntelliTest is largely the same as with the preview version as shown in this article.]

Adding Guard Clauses
One thing that I mentioned yesterday is that we could add code contracts to put limitations on the parameter ranges, and Smart Unit Tests would pick up on that. So, I actually tried it.

Now I didn't use the "Contract" class/methods because I'm still a bit wary -- the classes are built in to the .NET Framework, but you need to download an IL re-writer from Microsoft Research in order for it to work. I'm hoping the re-writer shows up "in the box" one of these days (or gets integrated into the Roslyn compiler).

So instead of using code contracts, I added a couple of guard clauses to the method:

The first statement checks to make sure that the "liveNeighbors" parameter is within the valid range of 0 to 8 (inclusive). Otherwise, we throw an argument out-of-range exception.

The next statement checks to make sure that the value of the "currentState" parameter is valid for the enum. As a reminder, enums are just integers underneath, so the compiler will accept any value for them. Because of this, we really should verify that we're getting a valid value.

And we saw this yesterday. When we did *not* have the guard clauses, Smart Unit Tests created a test to see what would happen with a variable that was out of range:

Tests *before* the guard clauses were added

Test #5 shows uses a "currentState" of "2". The valid values are "0" (Alive) and "1" (Dead). And because we didn't have any checks in our code, this returns a valid result (actually, an invalid result from a business perspective, but a valid result as far as the compiler is concerned).

Updated Smart Unit Tests
When we run Smart Unit Tests against the new method with the guard clauses, we end up with a different set of tests:

Tests *with* the guard clauses

We still have the same tests (in a slightly different order), but we also have some new tests (and one test with a new result).

Just like we talked about yesterday, Smart Unit Tests goes through every code branch and generates tests with parameters to exercise those branches. The same is true for the guard clauses.

So, if we look at Test #3, we see a test with "8" as the "liveNeighbors" parameter. This is edge of the range for our first guard clause.

Then in Test #5, we see a test with "9" as the "liveNeighbors" parameter -- outside of the range of the guard clause. And notice the result: we get an ArgumentOutOfRangeException, and the error message matches what is generated by the method.

Is throwing an exception really considered a "passing" test? Absolutely. This is the behavior that we expect for an invalid parameter. Here's the actual test method that was generated:

Notice that there is an "ExpectedException" attribute. Normally a test that throws an exception results in a failure. But if we tell the testing framework that we expect to get an exception, then it becomes a "pass" -- in fact, it will fail if no exception is thrown.

Test #8 tests the other end of the first guard clause. It uses "int.MinValue" for the "less than 0" condition. And this generates the same exception we just looked at.

Finally, Test #7 looks at the value of our enum parameter. Before we added a guard clause, Smart Unit Tests did generate a test for this case -- "currentState = 2", but our method was not set up to catch this and returned a valid value.

But with the guard clause in place, this test now throws an exception.

Verifying Expectations
So it's interesting to see how Smart Unit Tests handles guard clauses. Just like our other code, it generates tests to exercise those code paths. And if there are exceptions, then it generates a test that assumes the exception is expected.

Again, these tests tell us what our code *actually* does not what it is *expected* to do. After generating these tests, it is vitally important that we analyze all of the generated tests and results. What we may find is that there is a test that throws an exception where we do *not* want an exception to be thrown. Smart Unit Tests does not know the intent of our code -- only what it actually does.

So, if our tests throw exceptions, we need to double check to make sure that it is the behavior that we want. Otherwise, we need to update our code and try again.

[Update 11/17/2014: Peli de Halleux from Microsoft Research was nice enough to send some clarification on how exceptions are handled by default. We explore this in the next article: Smart Unit Tests (Preview): Exploring Exceptions.]

Smart Unit Tests and Legacy Code
As I mentioned yesterday, I'm still trying to figure out how (or if) this technology fits into my workflow. There is one place where Smart Unit Tests is very compelling: working with legacy code.

If I walk up to an existing code base that does not have unit tests in place, I can run Smart Unit Tests against the code to generate an expected baseline behavior -- this is what the code does today. Then I can save off those tests and run them regularly as I make updates or refactor the code. These tests would let me know if I inadvertently changed existing behavior.

That is a very compelling use case for this technology.

Now there are also some questions when thinking about this scenario. If a legacy code base does not have tests (and maybe it's a bit of a mess as well), then it may be difficult to create tests due to dependencies between classes.

The example method that we've been using to generate tests is *extremely* simple. And it lends itself well to unit testing. It does not have dependencies on any external objects or methods. It only acts on the parameters being passed in. And it does not modify any shared state that affects other parts of the code. Because of this, it is very easy to test.

But when we get to more complex situations that may require dependency injection or mocking, it's not quite as easy to generate tests -- especially if the code is not loosely-coupled. (It looks like I'll be running Smart Unit Tests against different types of code to see how it reacts.)

Wrap Up
It will be very interesting to see how this technology evolves. We saw here that we can add guard clauses to our method and appropriate tests are added to exercise those code paths. I'm going to keep thinking about ways that I can integrate this technology into my programming workflow. It seems like there are a lot of possibilities out there.

Happy Coding!