A Last Look at Smart Unit Test (Preview): Square Peg & Round Hole

I think that Smart Unit Tests is a really cool technology. And it's been interesting to explore different aspects of it. I really want this to work for me (and I'm even listening to the PEX team talk about it on .NET Rocks! as I write this). Unfortunately, I think I've come to the conclusion that it doesn't fit into my workflow.

Before I get to my reasons for this conclusion, let's look at another example that I was playing with earlier today.

[You can get all of the articles in the series here: Exploring Smart Unit Tests (Preview)]

[Update: Smart Unit Tests has been released as IntelliTest with Visual Studio 2015 Enterprise. My experience with IntelliTest is largely the same as with the preview version as shown in this article.]

Testing a Value Converter
For this exploration, I looked at a value converter. These are common in the XAML world (and one of the cool things about data binding in XAML).

Here's what a value converter looks like in it's native state:

This allows us to data bind a "DateTime" object to a UI element that is expecting a "Brush". Here's a screenshot that shows this in action:

The background color of each of our list items is databound to the "StartDate" property of our "Person" object. Based on the date (or more specifically, the decade), we show a particular color.

Now I didn't expect to be able to test this "Convert" method directly just because of the complexity of the parameters. So, I created a simpler method. This takes a "Person" as a parameter and returns a "SolidColorBrush":

When we run Smart Unit Tests against this method, we run into a bit of a problem:

The first test is a genuine failure: we are not checking for a "null" in our parameter. But the rest of our tests generate "InvalidProgramException".

Let's look at the detail of one of these items:

This shows that there is a problem when running the tests. When we click into this, we see that the problem statement is where we say "new SolidColorBrush()". I'm not sure why there is a problem here. And I'm also not sure what we need to do to get the Smart Unit Tests to run as expected.

Just to make sure I didn't create some type of run-time error in this code, I created a little console app to test the code. I added the appropriate assembly references (as I did with the class library) since the "SolidColorBrush" is part of the System.Windows.Media namespace.

Here's the console application:

And here's the output:

So, I know that the code itself works. This means that there might be a problem with Smart Unit Tests or the Visual Studio 2015 Preview or the way that it makes its calls into the various assemblies. Again, we're dealing with a preview version, and I expect these types of issues to be resolved prior to release.

A Simpler Method
Since it seems like there's a problem with the "SolidColorBrush" statement, let's remove that dependency. Instead of returning a "SolidColorBrush", we'll just return a "string" to represent an HTML color.

Here's the new method (I just picked colors at random; I didn't try to match them to the colors in the other method):

When we run Smart Unit Tests against this method, we get a better outcome:

This is a much better set of tests (without exceptions). As noted above, Test #1 is sending in a "null" for the parameter. Since our method does not check for null, it generates an exception. This is exactly what we expect, and we probably want to put a guard clause into this method to prevent this.

The tests generate real "Person" objects for the parameters. And as we can see, it's smart enough to generate objects that exercise each branch of our code.

Let's focus on the "StartDate" properties:

Here we can see that DateTime.MinValue and DateTime.MaxValue are both tested. The rest of our tests have random(ish) dates that fall within the decades that will hit each of our branches in the switch statement. So we have "1972-12-31" which will hit the "1970" branch; "1984-12-31" will hit the "1980" branch; "1992-12-31" will hit the "1990" branch; and "2000-12-31" will hit the "2000" branch.

So this is pretty cool. Even when we have an object with multiple properties as a parameter, Smart Unit Tests figures out what the important parts of that object are in order to exercise the code.

The Verdict
I really want to be able to incorporate Smart Unit Tests into my workflow -- primarily because it's such a cool technology. But ultimately, I don't think I'll be able to use Smart Unit Tests on a regular basis.

Testing Legacy Code
The most compelling reason for Smart Unit Tests is to create tests for legacy code so that we have a good idea of what the code is doing right now.

But here's the problem: If legacy code is difficult to understand (by a human), then it probably is not easily testable -- either due to large methods, shared state, tight coupling, or complex dependencies.

If the code is in a state where it has small easy-to-understand methods that are fairly atomic. Then I can probably understand what it's doing pretty easily. It would still be great to generate tests for this code. But in the world that I've dealt with, I've rarely seen code like this in a legacy code base.

The legacy code that I've dealt with is generally the "difficult to test" kind. In that case, I'm digging out my copy of Working Effectively with Legacy Code by Michael C. Feather's (Jeremy's Review). Smart Unit Tests are not going to be able to help me out there.

Testing Green Field Code
Okay, so maybe Smart Unit Tests won't work well with the legacy code that I deal with, but what about the green field code that I write?

That's really the blockade that I ran into yesterday. This is typical OO-based view model code. And Smart Unit Tests was not much of a help here. The dependencies are well separated (meaning, the view model and services are loosely coupled), and the class is testable (as we saw with the existing unit tests), but Smart Unit Tests was not able to easily create a set of tests here.

I would love to have those tests auto-generated so that I could fill in the gaps in my existing tests. But I do not want to go through the effort of creating a bunch of scaffolding so that Smart Unit Tests can do that for me. I feel that I'm better off spending my time enhancing my existing tests.

Where Smart Unit Tests Excels
Smart Unit Tests does very well with atomic, functional-style methods. We saw this when we started by looking at a method on Conway's Game of Life. But as mentioned previously, these methods are generally easy to test anyway (at least compared to tests that require specific starting states and ending states with OO).

Ultimately, C# is an OO language. It has a lot of functional features that have been added (and I really love those features). But we can't ignore its basis. If we want to do pure functional development, then we really should be using a functional-first language (like F#) rather than an OO-first language (like C#).

Wrap Up
Smart Unit Tests is a really cool technology. Unfortunately, it is of limited usefulness in my own workflow. I'll be keeping an eye on Smart Unit Tests to see how things change and evolve in the future. And I would encourage you to take it out for a spin for yourself. You may find that it fits in just fine with the type of code that you normally deal with.

I've been exploring different testing technologies and techniques for several years now. And I like to try out new things. What I've found through this exploration is a testing methodology that works for me -- makes me more efficient, faster, and more confident in my coding.

Be sure to explore different technologies and techniques for yourself. What works well for me may not work for you. So experiment and find what makes you most effective in your development process.

Happy Coding!