Reconciling Clean Code and the Startup Mentality

Last week, I was at That Conference, and it was an awesome week (more on that later). But it was also a bit awkward. On Wednesday, the keynote by Max Lynch (co-founder of Ionic) was about startups and getting functionality to the users (or potential users) as quickly as possible. It was a great talk about some key features of successful projects.

A big theme to this was "It doesn't matter what your code looks like." The awkward part is that I was speaking about Clean Code and the importance of caring about the readability of code later that day.

Fortunately, a lot of people showed up for my presentation, and we had a really great time together. But I did get an interesting question afterward from someone who worked at a startup:

How do I reconcile "It doesn't matter what the code looks like" with the principles of craftsmanship?

The developer believed in craftsmanship and readable code, and he was a bit distressed that his boss was saying that they had to do things fast and dirty. As a bit of a surprise to myself, I actually had a good answer to give (at least I thought it was a good answer):

Treat the version 1 product like a prototype.

Let's explore this a bit more.

The Importance of Moving Quickly in Multiple Directions
I understand the startup mentality of "it doesn't matter what the code looks like" -- but with full understanding of the consequences of that approach.

The problem is that we only have an idea to start with, and we're not even sure if anyone will be interested in it yet. So we put together just enough code to get things working. And a lot of times, we don't even get that far; we just create a landing page to see if anyone is interested in the product before we even write a single line of code.

And once we start coding, we cannot invest too much time or effort into this. If it is a complete flop, we discard the code and move on to the next idea. We don't know what we need at this point. We're really just exploring. Ultimately, we're working toward a minimum viable product, which is just enough functionality to be useful to someone.

Even then, we don't know what feature will really resonate with our users, so we have to be ready for unexpected responses and to move quickly in another direction.

Just a Prototype
The only way we can use this approach is if we treat this code as prototype code -- even if we are releasing it to the users. This is our exploratory code -- where we figure out what excites our users and differentiates us from the rest of the pack.

There Will Be Problems
When we write code like this, there will be problems. We will have bugs that are hiding in the code. We will have issues scaling for more users. We will reach a point where it is difficult to add a new key feature.

And this is the point were "it doesn't matter what the code looks like" really starts to cause us pain. The best thing to do is discard what we have and write the code for our real product based on the prototype.

Clean Code Does Matter
It's really easy to fall into the trap of thinking that clean, readable code doesn't matter. After all, if we can be successful without it, who cares?

This works short term, but things fall over in the long term.

Once we have our viable product, and we start coding "for real", we need to pay attention to craftsmanship principles. We have to make sure that we don't introduce bugs in our code. And we do this through automated testing (the proof that our code does what we think it does) and through readable code (if the code is easily readable, then the bugs have nowhere to hide).

In addition, since we now know what features make our application successful, we can leave spots for abstractions and extensibility points so we can easily add new features.

And this is where we start to think about the long-term viability of our product. If we treat our version 1 as the real version rather than the prototype, we're just setting ourselves up for a lot of pain. But if we take the learnings from the prototype to then create the actual product, we can reap the benefits of our quick, maneuverable process and use them to create a sustainable product.

Final Words
In the Wednesday keynote with Max Lynch, the focus was really on starting many companies (one after the other, and sometimes several at the same time) based on ideas that we don't know will work. We need to figure out what resonates with our users quickly so that we can focus our efforts.

So, my advice to people who are coding "fast and dirty" and don't feel right about the process: think about this as a prototype. It is an exploration of what might be useful. And just because we release that product to our users does not make it any less of a prototype. Expect to discard this code once we have gotten what we need to know from it.

It Does Matter What Code Looks Like
Ultimately, it *does* matter what the code looks like. Having readable code is what makes it easier for us as developers to make changes to meet our users needs. If we can't quickly meet our users needs, then we've failed. If we don't go back to replace that prototype code, then we'll end up falling over at some point.

If we create a mess so that we can get some learnings out of it, that's fine. But we can't leave things that way. For long-term viability, we need to think about the quality of our code which ultimately impacts the quality of the product.

Happy Coding!

Rewriting a Legacy App - Production Tests and Updates

A couple months back, I came up with a minimum viable product to replace an existing application (articles: Rewriting a Legacy Application). Even though it only took a few days to rewrite the application, I didn't put it into production right away.

Since there are some unknowns regarding the computer and hardware interaction, I wanted to make sure that I was around to watch the application once it was in production. Since I did a lot of traveling in November and we had holidays in December, I decided to hold off.

I finally put it into production late last week. In addition, I uploaded the project to GitHub and started tracking updates (GitHub project: https://github.com/jeremybytes/house-control). Time for a status update.

Commit Log
Here's the commit log for the changes that were made since the last article:

Let's walk through these (starting with the oldest item at the bottom).

Production Runner

The first thing I needed was an application that I could put into production. So far in the project, I had the libraries and a test console application (called "X10Test") but no production application.

I created a new console application called "HouseControlAgent". This is pretty simple. In fact, the Main method is simpler than our test application:

When we new up the "HouseController" object, it loads up the schedule and starts the timer. At some point in the future, I might want to add "Start" and "Stop" methods to this class. But for now, it will be tied to the console application.

This gave me something that I could run on the production machine. I copied files over, stopped the old system, and started up the new one.

Current Status
So far, things have been running pretty well. I haven't noticed any drop outs, and the schedule items seem to be activating at the right time.

Remove Test Records

The next thing I had to do was remove the test records that we had in the system. As mentioned in Part 4, testing things that are scheduled is a bit tricky. So we had some test records in place:

This is part of the "Schedule" object, which means that we don't have any external control over it. So the first thing I did was remove the test records from this class.

Moving the Test Records?
Then I thought about adding them back in somewhere else -- like in the console application. This would give us a quick way to verify that the system was working once we started it up in production. We would not have to wait for a schedule item to fire.

But there's a problem with this. The schedules are based only on the time portion of the DateTime object. That meant that if we added a schedule item at application start up, it would fire *every day* at that time. Probably not what we want.

So, I started thinking about what to do next. We'll loop back around to this, but first...

Removing the One Minute Window

Another thing that was mentioned in Part 4 is that we have a 1 minute window where we process scheduled items. Since our timer fires on 30 second intervals, this means that we end up sending each command 3 times.

This is the code that was in the legacy application. This application was in production for many years and ran on different types of hardware during that time. Sometimes the commands would not register with the X10 system, so I added this window that would send the command multiple times.

To see whether this was still necessary, I removed the window. I changed the timer to run on a 1 minute interval (instead of a 30 second interval), and I also compressed the window to 30 seconds.

Timer with 1 minute interval

30 second window

The result is that each command is now only sent 1 time.

Current Status
This has seemed to be working okay over the last few days. Unfortunately, I ran into a bit of a snag earlier today (just a couple hours ago). An item was scheduled for 4:30 p.m., but it didn't happen.

I checked the application, and it showed the command was sent at 4:30 p.m., but it was not registered by the system for some reason. I'll need to keep an eye on this. If things are not reliable, then I might need to go back to the expanded window.

Or I might come up with an entirely different solution where each command is sent more than one time. We'll have to keep an eye on things and see. That's one of the joys of working with this type of hardware. Since it's communicating with RF and through the power lines, there are multiple opportunities for interference.

One-Time Events

The idea of keeping the ability to put in test records turned into something a little bit bigger. I wanted a way to put in events that only fire one time -- this might be a good way to do a "status check" when the system starts up. We may even want to reset all items to a known state.

But as mentioned, the problem with the current schedule is that it was based on the time only (not the date). If I wanted to support these one-time only events, I would need to make a few updates.

First, I created a new enum "ScheduleTypes" that we could use:

Then I added this to the "ScheduleItem" object that we have. For anything coming from the "Schedule.txt" file, I just set things to "Daily" (but I'll want to be able to use the other options in the future).

In order to support adding these one-time events, I added a new method to the "HouseController" object:

This makes it easy for us to add one-time items in our console application.

Using the Date
But there's another consideration. How do I know when it's safe to remove a one-time event? If we're only basing things on time (and not date), then I don't know if this event has already passed or if it's still 23 hours in the future.

So I made some modifications to the system so that it takes into account both the time *and* date of a ScheduleItem. And we also have to treat our schedule a bit different.

For example, consider the items in our "Schedule.txt" file:

The dates for all of our items is January 1st, 2000. If the date portion is now important, we need to deal with this somehow.

And that's when I came up with the concept of "rolling" the schedule.

Rolling the Schedule
"Rolling the schedule" means that we take any items that are in the past, and re-schedule them for the next day. This way, all of our schedule items will have DateTimes that are in the future.

The method that does this is a little wonky right now, but here's what we have:

This loops through all of the items in our schedule, then looks for any items that are in the past.

From there, we figure out what to do with the item based on the schedule type. For example, if something is listed as "Daily", we want to roll the schedule forward to the next day. And we have a helper method set up to do this:

This will take today's date, add one day (to make it tomorrow) and then append the time from the event. So if an item was scheduled for "January 6th, 2015 at 3:15 p.m." (which is in the past at the time of this writing), it will be rolled forward to "January 7th, 2015 at 3:15 p.m." (which is in the future at the time of this writing).

I think this may run into some problems with midnight schedules; I'll need to do a bit more testing with this.

One-Time Events
But the important part of rolling the schedule is that is also lets us have our one-time events. Notice this code from the "RollSchedule" method:

This will remove an event that is marked as "Once" and is already in the past.

As a side note, since we're removing items from our schedule, we can't use a standard iterator and "foreach" loop. (We're not allowed to add or remove items in an active iterator.) So instead, we have to use a "for" loop.

And one other thing to note about our "for" loop is that we are counting backwards. We start at the bottom of the list and work our way up. The reason we do this is that we are potentially removing items. By going backward through the index, we make sure that we don't end up with an "index out of bounds" error.

Unit Testing
As you might imagine, there are a lot of things that can go wrong when we roll the schedule. Because of this, I put some unit tests into place. This will let us test the helper methods that we have to roll the dates and also confirm that the schedule object itself is working as intended.

ScheduleTests: GitHub: HouseControl.Library.Test/ScheduleTests.cs
ScheduleHelperTests: GitHub: HouseControl.Library.Test/ScheduleHelperTests.cs

I found these tests extremely helpful. As I was writing the tests and helper methods, I found a lot of problems with my code. And having the tests made it easier for me to figure out what I needed to do to get the behavior that I was expecting.

There's still more work to be done with the tests. I'll talk about this a bit more in the "Open Issues" section.

Current Status
This functionality is working fine so far. I can add one-time events that are scheduled at start up. Then after the events run, they are removed from the schedule.

Update 1: Just after completing this article, I discovered that the schedule items are not rolling properly on application start up. This is a problem with the "RollForwardToNextDay" method above. Things are getting rolled to "tomorrow" even though the initial schedule should be "today". This is now listed as "Bug" in the GitHub Issues. It looks like I have some more work to do on this...

Update 2: Issue is fixed with commit 956aab2. Solution was to use the current date when loading the schedule items from disk. Then the initial schedule roll works as expected. Details in the logged issue.

This functionality will also be good for future use. If we want to send one-time commands to the system (for example, we want to manually turn on a light), we can use the one-time event feature. This puts us in a good position with regard to that.

Open Issues
Since I have the project on GitHub, I thought that I would try out some of the tracking features for enhancements. I've opened three issues so far:

These items are there primarily to make testing easier. In addition, I know that there are some things that need to be improved with my handling of DateTime (thanks to the master of all things time-related: Matt Johnson (@mj1856 on Twitter).

The biggest reason I opened these issues is that I took an initial stab at this functionality and managed to get things twisted up pretty badly. This is where I was thankful for having a branch in Git that I could simply abandon to get back to my working code. These issues are my way of approaching things in smaller chunks.

Centralize "DateTime.Now"
Having DateTime.Now in the code makes things especially challenging to test. I've looked into several solutions, and I think the best one is to abstract out the calls to DateTime.Now so that they can be replaced for testing.

The first step for this centralize any calls to "DateTime.Now". I already have plans to create some helper methods (in the "ScheduleHelper" class) to do things like "check if this date is in the past" and "calculate how far away this event is from right now".

This doesn't immediately make things more testable, but it does set up the next step.

Abstract calls to "DateTime.Now"
Once things are centralized, it's easier to set up an interface for a TimeProvider that will give us the current time. With this in place, we can replace our centralized calls to "DateTime.Now" with calls into our TimeProvider.

Once we have this abstraction, we can create a mock or fake object that we can use for testing. The advantage here is that we can choose whatever time we want to represent the "current time". This lets us create tests for a lot of scenarios that would be difficult to test otherwise.

After this is in place, I expect the number of unit tests to balloon.

Update to UTC times
This last issue is for stability purposes. I've had enough conversations with Matt to know that dealing with local times is not the best way of doing things. There will be issues around Daylight Saving Time and other things that could come up.

If you want to take advantage of Matt's knowledge, you can check out his Pluralsight course "Date and Time Fundamentals". Unless you take a look at this, you may never realize just how complicated it is to deal with DateTime properly (especially when globalizing applications).

Wrap Up
I managed to get the MVP into production (which is good). I also got the project uploaded to GitHub. And I've already started working on improvements to the application.

Applications evolve over time based on the needs of the users. In this case, I have a pretty easy job: I already know what the user needs (because that's me). But as mentioned before, there are a few more features that I would like to add -- hopefully before the summer time when I need the air conditioner to go on/off on a reasonable schedule.

Happy Coding!

Rewriting a Legacy App - Part 4: Completing the MVP with Scheduling

It's time to wrap things up for the re-write of the legacy application (at least the immediate features). In Part 1, we looked at the existing home automation application and came up with a minimum viable product (MVP). As a reminder, here are the features that we need:

Needed for MVP:

1. Send commands through the serial dongle
   This is the purpose of the system.
2. Send commands for 8 devices on House Code "A"
   These are the only devices that are actively used.
3. Fire on/off/dim events on a schedule
   To maintain the air conditioner functionality (and current lighting schedule).

In Part 2, we built a test library to send commands through the serial port to the home automation hardware (thus fulfilling requirement #1). In Part 3, we looked at generalizing the functionality so that we could send commands to various devices (thus fulfilling requirement #2).

All that's left is to create a scheduler. Sounds pretty simple, but I also needed to create some wrappers and test objects along the way.

[Update: Code download and links to the entire series of articles: Rewriting a Legacy App]

Concealing Details
As mentioned in Part 3, I wasn't really happy with the way that I left the objects. Here's how we interacted with the SerialCommander and MessageGenerator:

Rather than having the application need to know the details of how to generate the message and then call into the commander, we'll wrap things up into a single class that handles those details. Here's what I want my application code to look like:

So, I created a HouseController object that wraps up the message generation and serial port interaction. This has a SendCommand method that takes a device number and a command. From the HouseController class:

This method uses the static MessageGenerator class to get the message, and then it uses the a private instance of the SerialCommander object (the "Commander" property) to interact with the serial port.

Faking Hardware Interaction
As I was going through creating these objects and moving code around, I ran into a restriction that I found to be a bit of a hassle. I needed to have the serial dongle plugged into my development machine in order for the code to run. If I ran the code without the dongle plugged in, I got the following error when calling into the SerialCommander object:

Since I was comfortable that the hardware interaction was working, I wanted to be able to continue to develop even when the hardware was not plugged in to my dev machine -- specifically, I wanted to keep it plugged in to the machine running the legacy application so that it would continue to function.

In order to do this, I created a simple interface: ICommander. Since the SerialCommander only has one publicly-exposed method, this interface is pretty simple:

With this interface in place, I could add the abstraction to the HouseController class (through the Controller property) and provide a fake implementation that didn't require the hardware.

The FakeCommander implementation is pretty simple:

It doesn't need to actually do anything, but I echo the message to the console window so that I could see that the method was getting called.

Inside the HouseController class, I set up the property so that we could easily use Property Injection to swap out a fake commander:

For more information on Property Injection, check out my materials on Dependency Injection (and specifically a blog article on the Property Injection pattern).

If we do nothing (that is, we do not set this property directly), then the first time it is used, it will automatically create an instance of the SerialCommander object. This is the default behavior that we would like to have in our production environment.

But for testing, we can override the property with a fake or test implementation. Here's our updated application code that injects the FakeCommander:

Notice that after we create the HouseController object, but before we call any methods on it, we set the "Commander" property to our fake object. This lets us run our application without actually interacting with a serial port.

Scheduling Data
What we really need to do is implement a schedule. For that, we'll need to be able to read the data from a persistent location (and in the future, add some UI that will make it easy to manage this data). I decided to keep things pretty simple. I created a ScheduleItem object that represents the data that we need:

And I created a Schedule object which is a collection of ScheduleItems. In addition, this object is able to load data from a CSV file from the file system. I decided on a CSV file because it is fairly simple, and I already had code that I could borrow from another project.

Here's what the schedule file looks like:

And here's what the loading code looks like (in the Schedule class):

There isn't any error handling here, so if the data is bad, we'll run into problems. But we're good for the MVP. We'll work on making this more robust as we need to.

Now that we have the schedule data, we need to execute it somehow.

Executing Scheduled Commands
I added the scheduling functionality into the HouseController class. I'm just using a Timer for this (the Timer is coming from System.Timers -- this seemed to be the best for my needs):

In this code, we set up a timer that is set to fire every 30 seconds. In the constructor, we hook up the event handler and start the timer. In addition, notice the "Schedule" property. This gets instantiated when this object is constructed (and the constructor code of the Schedule loads up the data from the file -- we'll look at this in just a bit).

When the timer fires, it runs the following code:

This code looks at the schedule items and filters the list based on items that are (1) enabled and (2) are scheduled to occur within 1 minute of the current time (I'll talk about this 1 minute window in just a bit). This calculation is done in the TimeDurationFromNow helper method:

This code is dealing with "TimeOfDay" because we want to ignore the date portion of the datetime values and just look at the times. The "Duration" method will give us the absolute value of our subtraction (so we'll always end up with a positive number).

Once we have a list of items that we need to process based on the current time, the scheduled time, and whether they are enabled, we just loop through them and call the "SendCommand" method for each one.

The last step in the timer event handler is to spit out some debug code to our console. This way we can check to make sure that our filter is running correctly.

Testing the Schedule
Testing schedulers is always fun. One way is to update the data file so that the scheduled time is in the not-too-distant future, then start the application and wait. This is not a fun way of doing things, so I cheat just a little bit.

Here's the constructor of our Schedule class:

In addition to loading the schedule from the file, I manually add 3 new schedule items for the not-too-distant future. This way, I don't have to wait very long, and I don't have to constantly update the data file.

Now, if we run our console application (and wait several minutes), we see the following output:

This may look a bit strange, so let's walk through this. The first 4 lines are coming straight from the Main method that we saw earlier. It outputs "Starting Test", turns on device #5, turns off device #5, and then outputs "Test Completed". Everything after that is from the Timer event running.

The first time the timer "ticks", it processes one record (our first test record from above). Remember that the timer fires 30 seconds after our application starts, and our first schedule item is set for 1 minute after the application starts. Since this is within the 1 minute window that we have defined above, that first command is run.

Then we have the output that shows us that 1 schedule item was processed, that we have 17 total items in our schedule (which includes 14 items from the file and our 3 items in code), and that 8 of the schedule items are active (in our file, the "Summer" items are marked as inactive).

If we look at the next set of records (from 5:35), we see that 2 schedule items are processed. The first command is sent (again) since it is still within the 1 minute window, and the second is sent since it enters that 1 minute window.

And if we follow this through, we can see all 3 of our test records go through the process.

Why the 1 Minute Window?
So you've probably noticed that we end up sending each command 3 times (30 seconds apart). This is a result of having the 1 minute window set up in our scheduler. But why do we need this?

This is based on my experience with the hardware. What I've found is that not every command gets registered by the system. I don't know if this is because the commands aren't received or if there is temporary interference in the power lines that prevent the transmission to the module. I just know that it is not 100% reliable.

Because of this, in the legacy application, I set up this processing window so that schedule commands would be sent multiple times. In practice this isn't a problem. If I send a command to turn off the air conditioner and it is already off, then nothing happens. And this is better than a command getting "missed" and a device staying on longer than intended (or not coming on as intended).

I'll need to do some long-term testing to see if this is still needed. But since the problems are intermittent, there's no way that I'll really know until the code is running for a while.

This Completes the MVP
The code isn't pretty (but it's not ugly, either). And there are still a lot of features that I'd like to add. But this completes our minimum viable product -- that is, a product that we can put into production to replace the current legacy system. Let's quickly review:

Needed for MVP:

1. Send commands through the serial dongle
   This is the purpose of the system.
2. Send commands for 8 devices on House Code "A"
   These are the only devices that are actively used.
3. Fire on/off/dim events on a schedule
   To maintain the air conditioner functionality (and current lighting schedule).

We have fulfilled all of these requirements. We can send commands through our serial dongle; we can communicate with 8 devices on house code "A"; and we can send commands based on a persistable schedule. So that's all we need!

More To Come
But even though this is all I *need* to replace the existing application, this is not all that I want from the system. I would like to have a UI to set the schedule (and enable/disable schedule sets). I also want to code up the "dimming" commands (I deferred this for the MVP since I decided it was not critical to initial implementation).

I would also like to have a network-aware interface so that I can send commands remotely. And I would also like something that would keep a record of the state of each device. This last one is a bit difficult to handle. There is no way for us to query the system, which means that we need to maintain state information ourselves. And there's the added difficulty that if someone turns a device on or off using the physical remote, there's no way for our application to know about that.

Wrap Up
This has been an interesting exercise to look at an existing application and work through implementing a minimum viable product. What I discovered was that the "minimum" really wasn't that much compared to what existed in the legacy system. There were a lot of features that aren't needed (but were nice to have) and some features that aren't used at all.

The result is a very small set of features that could be implemented very quickly. It took just a few days to get everything running. Being able to release software so quickly leaves us with a good sense of accomplishment (a usable product) and gives us motivation to keep moving forward with other features.

Running through these types of exercises are important practice for us. It gives us better perspective when we're dealing with our business users. There are a lot of things that are wanted, but we can focus in on one or two items that are really needed and implement them quickly. This makes our users happy. And ultimately, this loops back around to my musing on No Estimates and Partnering with the Business.

What goes around, comes around.

Happy Coding!

Rewriting a Legacy App - Part 3: Accepting Commands

So, I'm continuing to work on my house automation software to replace my legacy system. In Part 1, we looked at existing functionality and figured out our minimum viable product (MVP). In Part 2, we got the basics of the hardware interaction working. Now we'll expand things to work with various devices and commands.

But before we move on, let's review the requirements:

Needed for MVP:

1. Send commands through the serial dongle
   This is the purpose of the system.
2. Send commands for 8 devices on House Code "A"
   These are the only devices that are actively used.
3. Fire on/off/dim events on a schedule
   To maintain the air conditioner functionality (and current lighting schedule).

We've already got #1 in place, so let's expand things to #2 -- being able to work with all 8 devices on the system.

[Update: Code download and links to the entire series of articles: Rewriting a Legacy App]

Separation of Concerns
Now, I realize that the typical way of implementing a minimum viable product is to not really worry about the code organization (just get it working). I have trouble working that way. I like to clean code up as I go along.

When moving on to the next step of accepting commands for various devices, I really wanted to separate the concerns. One thing I didn't like in the test code was the mixing of building the message (specific to the X10 system) with sending the message (interacting with the serial port). So I created a new library and a couple of classes to handle this:

The first file is "DeviceCommands.cs". This simply has an enumeration to hold the "On" and "Off" values (and in the future, it will handle other commands as well).

The MessageGenerator is responsible for building the message that we'll send through the serial port, and the SerialCommander is responsible for sending those commands. Let's look at the SerialCommander first.

No Sleeping Allowed
When we were going through the test code in the previous article, it was littered with Thread.Sleep commands. These are generally not good because they block the current thread. But there's some good news for our code.

I did some additional tests and found that the Thread.Sleep commands were not necessary. When I removed them, the hardware interaction continued to work just fine. As mentioned, I brought these over because they were part of the legacy implementation. But they may have been needed on earlier systems just for timing issues. Perhaps (just a guess) modern computers run fast enough that they no longer run into issues here.

Creating and initializing the serial port is very similar to what we had in our test code:

The only difference here is that I added a configuration setting for the COM port. This is the one part of configuration that may need to be changed depending on the physical computer. The code also defaults to "COM3" (in case there is no configuration setting). This probably isn't a good way of handling a default, but we can loop back around to this later.

The method to send the command has a new parameter, but the bulk of the code is unchanged (except for the removal of the Sleep commands):

Just like before, it opens the serial port and initializes the DTR/RTS control signals. Then it loops through the message and sends the individual bits. And finally, it closes the serial port.

The biggest difference is that we're sending the message in as a parameter. We have to build this message somewhere. And that's the responsibility of the MessageGenerator.

Generating the Message
In order to fulfill requirement #2 for our MVP, we need to be able to interact with devices 1 through 8 on the "A" house code. To facilitate this, I created some static variables that we can work with.

I decided to go ahead and stick with strings for these values. I could have made them byte arrays and done shift left/shift right to process them, but the strings aren't causing any problems at this point, so we'll stick with them for now.

This code has the "header" and "footer" values that we saw earlier. In addition, I noticed that the first byte of the message (the middle 2 bytes) was the same for all of house code "A". So, I created a variable just for that. This means that our changing command portion is only 1 byte long.

To make things a bit easier, I created 2 lists: one for the "on" commands, and one for the "off" commands (I don't show the entire entry for the "offCommands" list, but you get the idea). This will give us a list that we can index into in order to find the value that we want. For example, if we want the "on" command for device #2, we can look at "onCommands[2]". I added the "dummy" values at the top of the lists since the indexers are 0-based. That way, we don't need to subtract 1 to get the right value.

With these static variables in place, we can now create a method that builds our message:

Notice that the "GetMessage" method takes 2 parameters: an integer for the device number and a DeviceCommands for the command. I spent a bit of time working on the parameters and the layout of the static variables so that this method would be fairly easy. Some of my other ideas included nested switch statements -- that seemed overly complex.

I start by validating that the device number is in a valid range. Since we're only supporting devices 1 though 8, this is important to validate.

Next, we start creating the message with the "header" and "codeA" parts.

Then we switch based on the command. If the command is "on", we look in the "onCommands" list from above; otherwise, we look in the "offCommands". As mentioned earlier, we can index into these lists based on the device number, and we'll get the right command string.

Finally, we add the "footer" to the message and return it.

Running the New Methods
Now that we have these classes in place, we can use them to send commands to any of our connected devices. Here's our updated console application (from Program.cs):

This uses our new classes to turn on device #5 and then turn off device #5. I was using this device since it was a lamp in the room that I was working in. This let me verify visually that the commands were working -- although I think my cats were getting kind of annoyed that the light kept going on and off.

And, of course, we can change the parameters of the "GetMessage" method to interact with any of the devices on the system.

Closer to MVP
Let's review our requirements again:

Needed for MVP:

1. Send commands through the serial dongle
   This is the purpose of the system.
2. Send commands for 8 devices on House Code "A"
   These are the only devices that are actively used.
3. Fire on/off/dim events on a schedule
   To maintain the air conditioner functionality (and current lighting schedule).

By creating a parameterized method (along with the static variables for the various commands), we've fulfilled requirement #2. We can now interact with the 8 devices that are on house code "A".

I'm not completely happy with this implementation. In the console application, we need to create both the SerialCommander and the MessageGenerator objects. I don't want my calling application to have to know so much about the details. So at some point, I'll wrap these up into a class that is easier to work with.

What's left is to be able to process a schedule (requirement #3). This will be a little more involved. We'll need some sort of class that wakes up at particular times to fire events, and we'll need some way to persist the schedule to disk. We'll be looking at these in future articles.

So we can see that we've made quite a bit of progress toward our minimum viable product. And I like the direction this is going in so far. By eliminating the features that we don't really need, we've managed to come up with a system that is pretty quick to implement.

Stay tuned for more articles on this. The scheduling portion should be interesting (but I may swing off into some other articles in the meantime).

Happy Coding!

Rewriting a Legacy App - Part 2: Working with the Hardware

The first step to rewriting an application is to figure out what we need (this is true when we're working on new applications, too). Last time, we looked at existing features and figured out the bare minimum that we would need to implement to replace existing use.

Note that I said existing *use* and not existing *functionality*. These are two different things. Here are the features that are currently used:

Needed for MVP:

1. Send commands through the serial dongle
   This is the purpose of the system.
2. Send commands for 8 devices on House Code "A"
   These are the only devices that are actively used.
3. Fire on/off/dim events on a schedule
   To maintain the air conditioner functionality (and current lighting schedule).

Step one is to interact with the hardware by sending messages through the serial port. I wrote a test application to make sure that this would work. The code isn't very good, but it gets the job done. We'll clean up the code as we go.

[Update: Code download and links to the entire series of articles: Rewriting a Legacy App]

Communicating with a Serial Port
The first piece of the puzzle is to communicate with a serial port. As a reminder, we have a 9-pin dongle that we hook up to a USB port with an adapter:

Us old-timers (well it wasn't really that long ago) used serial modems to connect to other computers (whether to Wildcat BBSs or other systems). So, we're used to the things that need to be set up for serial port communication -- things like baud rate, parity, number of stop bits, and so forth. If you had the wrong settings for the system you were trying to connect to, then it didn't connect.

When setting up a serial port in .NET, we need the same information. Fortunately, I can bring these settings over from the legacy system -- with one exception. The COM port will vary depending on the hardware.

To figure out the COM port, I just plug in the USB adapter and take a look at the device manager:

This shows that we're using COM3. With this information, we can initialize a serial port.

A Test Library
For the initial test, I created a class library that we can use to send commands to the serial port. I'm expecting that we'll do a lot of refactoring of this code along the way. I called the class "X10Utils", and it has the following initialization code:

I won't go into the details. The only thing that has changed over the years is the baud rate. This started out as a slower value when I first put the legacy project into production. But now, we basically max out what is available with the serial port.

Sending a Message
This is where things get fun. In order to send a message to the system, we enable and disable DTR and RTS. DTR stands for "Data Terminal Ready", and RTS stands for "Request to Send". These are serial port control signals, and they (and others) would determine if a system was listening, sending, waiting, etc.

But these meanings have nothing to do with how they are used here. The use here was decided by whatever engineer designed this interface to the dongle.

So, we enable/disable DTR and RTS in a particular order to send a message. Here's an example of two messages:

The first turns on device A01 (that is device #1 on house code "A"). The second message turns off that same device. I represented the messages as "1"s and "0"s because that's how they were represented in the documentation that I originally used.

For each "1" in the message, DTR needs to be disabled and re-enabled. For each "0" in the message, RTS needs to be disabled and re-enabled. These sequences get translated into an RF signal that makes it's way to the transceiver module (don't ask me how that happens -- I'm not an electrical engineer).

Each message is broken up into 3 parts. The first line (2 bytes) is the header. This is the same for every message. The last line (1 byte) is the footer. This is also the same for every message. The middle line (2 bytes) is the actual message that is being sent.

Strings?
You've probably noticed that the message bits are represented as strings. I don't think that this is the best way to represent this data, but this is how it was represented in the legacy application. I left it for now since it's easy to copy over the data from the legacy application. Here's a small sample of the data:

You'll notice that the "xA1on" and "xA1off" values correspond to the middle lines of the code sample above. As you might imagine, there's quite a bit of this (16 devices and 16 house codes). This is one reason why I'm limiting things to just the devices being used (#2 from our MVP requirements).

Preparing the Message
There's really 3 parts to sending a message: (1) Prepare for the message, (2) Send the message, (3) Reset when done.

Here's the code to prepare to send the message:

This method takes a parameter to decide whether we're sending an "On" message or an "Off" message. The "message" variable is set to the correct value based on the parameter. (Again, not great code at this point.)

Next, we open the serial port and enable both DTR and RTS. The initializes the hardware device.

Then we have a "Thread.Sleep(10)". Now, Thread.Sleep is usually a bad idea. And one of the first things I want to do is to change this. But to get started, I'm moving the working code over from the legacy system. When I first wrote this code, I found that I needed "Sleep(100)"s in place between processing each bit. And I found that this value would vary depending on the computer. On previous systems, this value has been "100" and "50". I found that on my current computers (which are extremely fast compared to the hardware this was originally running on), I have the value down to "10". We'll work on getting rid of the Sleep a bit later.

Sending the Message
Now that things are prepared, we need to send the actual message. Here's how I process our "message" variable:

I simply "foreach" over each bit of the message. Since these are strings, this comes out as the character "1" or the character "0". Based on that, I click off RTS or DTR.

Again, the "Sleep" values are in there to give things a chance to register.

Resetting Things
After sending the message itself, we reset the serial port:

This closes the serial port once we are done sending the message.

So, there's not all that much code here. Let's see if it works.

Testing the Code
The best way to test this method is to plug things in and run the method. I created a console application to do just that.

This is pretty straightforward. I create an instance of our library class and call the test method twice -- once with "true" to turn the device on and once with "false" to turn the device off.

The main reason I'm doing both commands is because I'm lazy. The Transceiver Module (which is the one we talk to when we talk to the "A01" device), makes a satisfying clicking noise when it changes state -- either clicking on or clicking off. Since I run it through both states, I know that I should expect to hear 1 click (if it's already "on") or 2 clicks (if it's "off"). By sending both commands, I know I'll hear at least 1 click each time.

And when running the application, I do hear the module turning on and off: SUCCESS!

Where Are We?
So this is a really good start. We know that we can interact with the hardware. And this really is a big success. When I was first programming against this hardware, trying to figure out the timings and settings through trial and error, I went through about 2 weeks of frustration. Eventually I found out that I had my "1"s and "0"s inverted. Hearing that first click and seeing a light go on was a huge step forward.

Let's review what we need for our minimum viable product.

Needed for MVP:

1. Send commands through the serial dongle
   This is the purpose of the system.
2. Send commands for 8 devices on House Code "A"
   These are the only devices that are actively used.
3. Fire on/off/dim events on a schedule
   To maintain the air conditioner functionality (and current lighting schedule).

We've accomplished #1: interacting with the hardware. As we move on to #2, we need to do some thinking and refactoring. We'll need to figure out if storing the commands as strings is practical (maybe it is). There may be a better way, but we also may not care about that when we're coming up with a working product. It may be better to spend our time on #3, which is a bit more complicated.

I haven't posted the source code because there really isn't much that you can do with it without the hardware. But if you'd like to see a copy, let me know, and I'll be glad to make it available.

We're well on our way to creating a replacement for an existing application. And it doesn't look like it will take much more work to get the minimum viable product.

Happy Coding!

Rewriting a Legacy App - Part 1: Looking for the MVP

Okay, so I've been meaning to re-write an application that has been running in my house for 10 years. The original was written in Delphi (which I haven't used for new projects since 2005, so it's been a while), and it's been serving it's purpose. But there are some features that I'd like to add, and some that I don't really need. So, it's time to analyze the existing application, come up with a minimum viable product (MVP), and move forward with a re-write.

[Update: Code download and links to the entire series of articles: Rewriting a Legacy App]

Home Automation Hardware
The application is to control home automation hardware that uses X10 technology. This is based on a starter kit that my brother gave me for Christmas one year. It came with a serial dongle, and he thought I would have fun programming for it.

The technology uses power line communication. Here's the basic hardware:

This shows the Transceiver Module (the one on the left with the antenna). This receives RF transmission from a physical remote (not pictured) or from a serial dongle (lower center). When it receives the command, it transmits to the other modules through the power lines in the house. On the right is a Lamp Module -- this has a built-in dimmer. There are also Appliance Modules that have on/off (but no dimming), and I have a special module for my wall-unit air conditioner.

The last piece in the picture is the serial adapter cable (from DB-9 to USB). This is so that we can hook up the dongle to a modern computer -- when was the last time you saw a serial port on a computer?

And if you're curious about how you program against the dongle, you enable/disable RTS and DTR in a particular order. I'll go into the details of this in a future article.

Original Intent
The primary reason I was interested in this technology is because I live in an apartment with a wall-unit air conditioner, and I wanted to be able to schedule the air conditioner to go on and off during hot days (when I wasn't home) to keep the place a reasonable temperature.

Here's the main screen of the current application:

This shows the various modules in my apartment (yeah, it's not a huge apartment, but it's the right size for me and the cats). This has buttons to manually turn things on and off. And the sliders are there for the dimmable items.

But the important part is the schedule:

This lets me set up a schedule to turn items on and off throughout the day. I can set the time, the function (on/off), the level (which is the dimming level), and the days of the week that this applies. On top of that, I have labeled Schedule Sets that let me enable/disable groups of commands.

For example, there is a schedule set called "Summer" that I can enable when it's hot. This turns the air conditioner on an off throughout the day -- currently set to alternating between 30 minutes on and 60 minutes off throughout the afternoon. I've found that this is enough to keep things a reasonable temperature. (And I'm sure the cats appreciate it.)

The user interface for the application isn't great, but this was created during the "battleship gray" era, so it's actually pretty impressive taken in that context.

Stuff That I Don't Need
The X10 system has the concept of a "House Code". This is the letter setting that you see on the modules. Each letter can accommodate up to 16 devices (the number on the Lamp Module). This lets you either segment devices if you have a lot of devices, or use a different setting if there is power line interference.

When I coded up the application, I coded for all of the House Codes. I made this configurable in the settings screen:

In reality, I've only ever needed House Code "A". So, I can eliminate the other settings (at least for now -- I can always add them in if I find that I need them in the future).

The other thing is that I coded for 16 devices. I have a total of 8 modules, and since X10 technology has gone out of style, I probably won't be acquiring any more (plus, I don't even use all the ones I have most of the year -- Christmas lights get put onto the system in December).

In addition, I don't really use the manual buttons in the UI. This application runs on the entertainment center computer. This computer is on all the time and plays music through the stereo (in addition to controlling the lights). The monitor for this computer is the television, and the only time I really use the application is to enable/disable the schedule sets. The rest of the time, I use the physical remote for manual operations.

Features I'd Like
There are a couple of features that I'd like to add. The first is being able to interact with the system over a network. The current application has a little bit of this -- it opens up and listens on a TCP port, and an application on the same network can communicate with it using a proprietary message format (nothing makes it outside of the local network).

Something else I'd like to add is temperature-aware operation. So, I'd like to be able to have the air conditioner cycle based on temperature rather than simply based on a schedule. For real-time operation, I would need some additional hardware (a temperature sensor), but I've also thought about hooking it up to a weather service. This way I would not have to manually enable/disable the schedule; I could have it automatically enable if the temperature is forecasted to be over 90F.

Finding the MVP
In order to replace the current system, I need to figure out the minimum set of features that I actually need.

Needed:

1. Send commands through the serial dongle
   This is the purpose of the system.
2. Send commands for 8 devices on House Code "A"
   These are the only devices that are actively used.
3. Fire on/off/dim events on a schedule
   To maintain the air conditioner functionality (and current lighting schedule).

Not Needed:

1. Manual control buttons/sliders
2. Map of the apartment
3. Scheduling UI
4. Configuration UI
5. TCP listener

So, what I'm saying is that in order to produce a minimum viable product that I can put into production, I don't need any of the current UI or networking elements. Now I'm sure that I'll want to add these later, but they aren't required to get started.

Not Much Needed
What we find is that when we analyze the current application and look at how it is *actually* being used, we really don't have very many features that we need to code up. As I've mentioned, I do have other features in mind, so I want to make sure that I don't code myself into a corner along the way. But let's start simply and work our way up from there.

I've already done some tests to confirm that I can communicate with the serial dongle through .NET (that's the environment we'll use for this application). So, it shouldn't take very long to implement these initial features.

The main reason that I'm writing about this is to motivate myself to actually do this project that I've been thinking about for a long time. I don't expect that the specific technologies will be of much interest, but the process of building the application will be interesting.

Next time, we'll take a look at the basic operation of sending codes through the serial dongle. Then we'll look at how we can use this to move toward the minimum viable product.

Happy Coding!