TDD Practice: Parsing Data from a Service

Time for a bit more TDD coding practice. As mentioned in previous articles (including this one), I'm not always sure how the test-first process will work for me when I'm coding up certain types of libraries. So, I decided to work on this some more by re-coding a library that parses data from a JSON service.

To get a full view of the articles related to this application, check here: Rewriting a Legacy Application.

The code shown here is available on GitHub in the "sunset-tdd" branch: jeremybytes/house-control/tree/sunset-tdd.

Original Approach
When I first approached this code, it was a bit ugly. I needed to do 3 things:

• Call the service
• Parse out the string for sunset time (which is UTC)
• Convert to a DateTime that represents local time

And this is the approach that I took when I built the original library. The original code was functional, but it was not in good shape.

Then I tapped into one of friends (who's an expert in all things date/time related), and he helped by cleaning up my time processing.

Then I split out the concerns a bit better and created methods to handle the various steps. This (and a little further refactoring) got the library into a fairly maintainable state. If you're curious about where things ended up, the file is available here: SunriseSunsetOrg.cs.

But I wondered if I could get there faster if I used TDD in the process. So, let's find out.

Breaking Things Down Differently
Granted, I have a bit of an advantage here since I've already written and refactored the code. But I'm hoping this will help me get a grip on how I can use TDD in similar situations in the future.

As a reminder, we want to use the Red-Green-Refactor approach. First we write a failing unit test (Red), then we write just enough code to get the test to pass (Green), then we clean up the code if needed (Refactor).

My main roadblock for this library was thinking that I needed to worry about calling the service first. But we'll put this on hold for a while, and jump to Step #2: parsing the JSON result.

For this, I created 2 new classes: "SunsetTDD.cs" (in the HouseControl.Sunset project) and "SunsetTDDTest.cs" (in the HouseControl.Sunset.Test project).

Note: Final versions of the files can be found here: SunsetTDD and SunsetTDDTest.

Let's start by reviewing the data that comes back from the service. Here's a service call with "February 15, 2015" as the date:

Test #1
To get started with the first test, we'll create a string of this sample data. Here's the sample data and the shell of the first test:

The first thing that we'll do is check the "status" of the service call. Here's our test filled in:

Since we're doing our tests first, we don't yet have a "CheckStatus" method for us to call. One thing I assumed was that "CheckStatus" was going to be a static method (notice the way that it is called) and that it would take the JSON string as a parameter.

One of the nice things about Visual Studio is that it tries to help us out as much as possible. In fact, if we click on "CheckStatus", we'll get a pop-up to generate the method:

(I also like to use the keyboard shortcut "Ctrl+." to get this to pop up.) When we select it, it creates a new method for us in our SunsetTDD class:

It's not very useful to start with, but at least now our test will compile. Now we have a failing test (Red).

Now, according to the TDD philosophy, we should write just enough code to get the test to pass. Here's the bare minimum:

But this is a bit disingenuous (and not very realistic). One thing to note here: I have Visual Studio 2013 Ultimate, which comes with a tool code called Code Lens. This is the small line that we see above the method that includes several pieces of information. The one we care about is the "1/1 passing" which tells us that there is 1 passing test for this method out of 1 total tests for this method.

Let's be more realistic and write a bit more code:

This actually deserializes the JSON object and checks the "status" property. And we can see that our test still passes.

Test #2
Now we'll go on to check the error condition. Notice above that in addition to the valid "sampleData" in our test file, we also have an "errorData" object. Let's use this in a test.

This test passes without us needing to make any updates to our code (note the green icon in the Code Lens information) -- and that's because we wrote the realistic version of the method rather than coding the absolute minimum.

Refactoring
Before going on, let's do a bit of refactoring. When we let Visual Studio create the method for us, it used "sampleData" as the name for our parameter. That's fine for the test, but not for our production method. So, we'll update our parameter name to "serviceData":

Parsing the JSON Result
Now that we know whether we have a valid result from the service, let's try to get a string value out of it. Here's our next test.

Test #3

Just like before, we are calling a method that does not yet exist: "ParseSunset". So, we'll let Visual Studio generate this method for us as well. Then we'll fill in our code:

This looks pretty good. And it uses a dynamic object so that we don't have to create an explicit CLR type that matches the JSON layout. But notice that our test is still failing (that's a nice thing about Code Lens: it shows the failing tests right with the affected method).

Let's click over to the test and check the message. We can click on the icon in Code Lens to get the details:

This tells us that our dynamic call failed somewhere. If we look at our data again, we can see that the property should be "results" with an "s". So let's update our code:

Now we have a passing test.

Test #4
Now that we've tested the "happy path", let's add a test that expects bad data.

This expects a "null" result if we pass in bad data. But we are currently "Red" (failing test), so we know that's not the case. Let's look at the failing test message to get some further insight:

This is actually the same error that we got before. That's because our code is still trying to get at the "results" property of our dynamic object, but the "errorData" doesn't have a "results".

So let's fix up our method to check the status before parsing:

Now we have a passing test.

Test #5
We'll do one more test on this method. We've checked to make sure that we get a string back if we have valid data, and we've checked that we get "null" back if we have bad data. Now let's check that the string we get back has a value that we expect.

The "expected" value is copied out of our sample string. And this is the value we expect our method to parse out. And as we can see, this is "green", so it is working as expected.

So that takes care of parsing out the UTC time string from our JSON data. Now we'll move on to processing that string value.

Getting the Local Time
For Step #3, we're turning the string time (which is a UTC value) into a date/time object that represents the local time.

Test #6
First the failing test:

So, based on a specific UTC time string and the current date, we want to get a date/time object that reflects those values.

As before, we've referenced a method that does not yet exist: GetLocalTime. We'll let Visual Studio create this for us and then fill things in:

This is the code that Matt Johnson contributed to create the local date/time object based on a UTC time string.

And we can see that we have a passing test.

Test #7
Now let's test an error state: what happens if we pass in a "null" for the time string. This could happen based on the return value that comes back from the "ParseSunset" method above. The value that we expect in this case is the date portion of the "currentDate" parameter that we pass in.

Notice that we have a failing test. Let's check the error message:

We end up with a null reference exception when we call "DateTime.Parse" in our method. So let's add a guard clause that does what we want:

Now if we pass in a "null", we get the date portion of the "currentDate" parameter -- just what our test expects. So we have another passing test.

Note: This may not be good behavior here. We may actually want to throw an exception instead. But this is the way that we've coded up the other library. This is subject to change based on a review of the error handling code in the application.

Sunrise Values
Now that we're successfully parsing sunset values, let's do the same for sunrise values. For this, we'll start by copying some tests we already wrote.

This is a bit of a jump (putting multiple tests in at once). But we know that we'll need these, and that the method we'll implement ("ParseSunrise") is going to satisfy all of them. I don't have a problem taking a bit of a shortcut here.

We'll let Visual Studio create the method, and then we'll fill it in similar to "ParseSunset":

And as we can see here, all 3 of our tests are now passing.

Implementing the Library Interface
We want to be able to use our "SunsetTDD" class in our live application. This means that it needs to implement the "ISunsetProvider" interface. So, we'll add the interface and then look at how we'll test it.

This has our interface with its 2 methods: GetSunset and GetSunrise. We'll fill these in using TDD. First, let's create a test.

Test #11

Since our method "GetSunset" takes a date/time and returns a date/time, we'll create some variables that we can use as parameters. Then we'll create an instance of our class and call the method.

This test fails since we haven't implemented the "GetSunset" method. We'll start with some minimum code:

Notice that we're using our sample data in this method rather than making a call to the service. We'll need to fix this in a bit. But this will get us pointed in the right direction.

After we have our sample data, we'll run it through our other two methods: ParseSunset and GetLocalTime. This results in a passing test.

Leaving a Spot for the Service Call
But we have a bit of a problem here: our data is hard-coded. We want to be able to get the data from a service and also be able to test the method without calling the service. Because of these requirements, we'll use some dependency injection.

For this, we'll create an interface:

The "GetServiceData" method will call the service in the production class. But we will mock this up for our tests.

To use this in our class, we'll create a field to hold the "ISunsetService", and we'll populate it with a constructor parameter:

This is a classic example of constructor injection. For more information on DI and constructor injection, check out "Dependency Injection: A Practical Introduction".

Updating the Test
Since we added a parameter to the constructor, our test is no longer valid:

Let's create a mock service that we can use as a parameter. For this, we'll bring in Moq with NuGet:

This lets us do cool stuff like this:

What we've done here is create a mock of our "ISunsetService". The "Setup" method specifies that if someone calls the "GetServiceData" on our mock object (with any parameter), we want to return our "sampleData" object that has our test data.

Then we pass "serviceMock.Object" to our constructor. When we say ".Object", it pulls the "ISunsetService" out of the "Mock<ISunsetService>" object.

Adding a Production Service
Now we can add a production "ISunsetService" object. Since this is making an actual service call using an HttpClient object, I can't think of a really good way to unit test this. I know that's a short coming, and I'm sure there's a way that we can do this. But for now, we'll just pull over the service code that we had in our other class library:

A Bit of a Problem
So this gives us a production service to work with, but we have a bit of a problem as far as our application is concerned. Where we use the "ISunsetService" in our application, we'd also need to create an instance of our "SunriseSunsetOrgService" object to pass in to the constructor. This isn't all that great.

A better idea is to use property injection. This will let us use the production service by default and still give us a spot to swap it out for unit testing. This is what we did previously when we created a time provider that we could swap out for testing.

Instead of the constructor, we're injecting the dependency through a property:

If we do nothing, then the production service ("SunriseSunsetOrgService") will be automatically created. But in our unit tests, we can assign our mock object to this property to use our test data.

One other thing we need to do is make sure that we reference the property (rather than the field) in our code:

Updating the Test (Again)
Our test now gets a compiler error (since we removed the constructor parameter):

It's easy enough to update this. We'll use the default constructor and then assign our mock object to the property:

Now our test passes, and things look pretty good for our library.

Testing the Production Service
We have passing tests, and we're pretty sure that our code does what we want. But we still need to make sure that the production service gets called as expected.

For this, we'll update the console application that we have in the "X10Test" project:

We've simply swapped in our "SunsetTDD" class as our sunset provider. And we do *not* set the property, which means it will use the production service by default.

And if we run the test application, we see that we get the expected result:

Notice that our sunset time is a little different from our sample data. That's because this is the sunset time for February 16th (as opposed to the 15th in our sample data). This is how we know that we're hitting the service rather than simply using our hard-coded data.

Wrap Up
So we've managed to recreate the functionality of our library by using TDD. There are still a few gaps: we haven't created tests (or filled in the method) for "GetSunrise", and we don't have any tests for the code that hits the service. But we have good testing and logical layout for all of our other methods: checking status, parsing sunrise/sunset from the JSON object, and turning the UTC times into local date/time objects.

[Update Feb 16, 2015: I added the tests/implementation for "GetSunrise" and also added caching. These are in the "sunset-tdd" branch in GitHub. The steps are described here: More TDD Practice.]

The code is much cleaner than when we took a shot at it without TDD. We could add a bit more functionality (which I may do in a later article). The big example is the cache. Our other library kept a copy of the data for a particular date based on the assumption that the same data may be used over and over again during a particular day. We could add that caching functionality to the new library as well.

I probably won't put this library into production. The current library is working just fine. But this was a really good practice for me using TDD. It gave me a bit more perspective on the approaches that we can take when looking at a problem. And I think that it will be useful during future explorations.

Coding practice is an important part of being a good developer. It lets us try out new techniques and experiment with things that might not go as planned. Then when we are building actual applications, we can use that experience to code more quickly and effectively.

Happy Coding!