Still More Smart Unit Tests (Preview): Exploring Exceptions

Last week I started an exploration of Smart Unit Tests (preview). The articles are collected here: Jeremy Bytes: Exploring Smart Unit Tests (Preview). In the last article, I added guard clauses to verify parameters to see how Smart Unit Tests reacts to it. I noted that the tests that generated the exceptions were still passing tests because they were annotated with the "ExpectedException" attribute.

I got a bit of clarification from Peli de Halleux (who is on the Pex team from Microsoft Research) about this behavior.

@jeremybytes Pex assumes that an ArgumentException coming from the type under test is expected; otherwise not
— Peli de Halleux (@pelikhan) November 18, 2014

This makes sense. Since Smart Unit Tests creates tests based on parameters, it makes sense that it treats ArgumentException different from other exception types since ArgumentException is likely to be thrown by things like guard clauses and code contracts that verify parameters.

Let's explore this a bit more.

[Update: Smart Unit Tests has been released as IntelliTest with Visual Studio 2015 Enterprise. My experience with IntelliTest is largely the same as with the preview version as shown in this article.]

Something Other Than ArgumentException
I'll start by changing one of the guard clauses to a different exception type.

I changed the "ArgumentException" to "IndexOutOfRangeException". Now let's re-run Smart Unit Tests.

Now we see that we have a failing test. Test #7 uses "2" as the "currentState" (and as a reminder, this is outside the range of our enum).

And we can see that the test does throw an "IndexOutOfRangeException". But this time, it treats this as a failing test.

Here's the detail of the test when we click on it:

Notice the attributes: we have an attribute "PexRaisedException" which notes the exception that was raised, but we do not have an "ExpectedException" attribute in this case.

So what we can see here is that if the code throws something other than an ArgumentException (or something that inherits from it), the result is a failing test.

Now if we were not expecting this exception, this test gives us *great* information. It tells us we need to guard against the exception or catch it and make sure we handle it cleanly. So, this can give us some great insight into scenarios we may not have thought about.

But that's not what we have here. This exception is perfectly fine for this block of code.

Allowing the Exception
So we have a problem: we really want to allow this exception. (Side note: in a real application, I would keep this as some subtype of ArgumentException, but we're just using this as a sample case for an exception that we expect in our code.)

Fortunately, Smart Unit Test allows for this. If we right-click on the failing test, we get a number of options:

These options are also available in the tool bar. Let's see what happens when we choose "Allow" and rerun Smart Unit Tests.

Now our test passes (Test #7). Here's what we see when we click on the details:

The attributes have been updated. Now we *do* have an "ExpectedException" attribute that says we are expecting an "IndexOutOfRangeException".

A New Test Project
When we choose "Allow" on our test, that setting needs to be saved somewhere (this is also true if we select one of the other options: Save, Fix, or Suppress). These end up in a new test project that gets added to our solution automatically.

The project is called "Conway.Library.Tests" (since it is based on the "Conway.Library" project). And the test class is "LifeRulesTest" since it is based on the "LifeRules" class.

Here's what the class looks like:

Since we chose "Allow", there are some attributes that reflect that.

Now, as Peli noted, "ArgumentException" is allowed by default. This is the first "PexAllowedExceptionFromTypeUnderTest" attribute. And based on the attribute, we can see that in addition to "ArgumentException" we also allow any exceptions that inherit from that exception (so things like "ArgumentOutOfRangeException" and "ArgumentNullException").

We also have a note that "InvalidOperationException" is allowed as well. We have the option of changing these attributes if we want to.

And then we move down to the test itself. Notice that it is attributed with "PexAllowedException" for "IndexOutOfRangeException". This means that we will no longer get a "fail" if that type of exception is thrown for this test.

Peli was also nice enough to point me to a wiki with more details from the Pex project (which is what Smart Unit Tests comes out of). You can a description of the various attributes here: Pex - Automated Whitebox Testing for .NET : Allowing Exceptions.

Wrap Up
So we can see that Smart Unit Tests gives us control over what exceptions we consider to be acceptable in our code.

I've been playing a bit with the other options that we have on the tests as well (Save, Fix, and Suppress). "Save" is pretty obvious -- it saves off a copy of the tests so that we can re-run them later and even modify them if we want. "Fix" allows us to add manual code to places where Smart Unit Tests may not have been able to generate a complete test case. I'm exploring this right now, so you can look forward to more info in the future. I haven't used "Suppress" yet, but I'll give that a try as I continue my exploration.

Look forward for more to come.

Happy Coding!