Programming by Trial-and-Error

I guess I must be getting old because I keep looking back at my history as a programmer to see what I can learn about myself and what works and what doesn't work. This week's realization is that I've been a programmer for a lot longer than I thought.

When I've written about my history, I usually mention that I wrote my first program on a knockoff Apple ][ way back in 1985. But that's not accurate. I wrote my first program long before that.

Christmas 1979
Way back in 1979, I got an awesome toy for Christmas: Big Trak. This was not a remote-control vehicle; it was a programmable one.

Merry Christmas!

It had a keypad on the back that you would use to punch in instructions (up to 16 of them). Then you press "Go", and it follows them (sort of) exactly.

It could move forward or backward in the very precise unit of "vehicle length".

It also could turn left and right. The units for this were a little easier: "15" was a 90 degree turn. It doesn't sound very intuitive, but there were 60 units in a full circle, which meant that it was similar to a clock face (we used analog clocks back then).

Finally, there was a "Fire" button that would fire off the blue "laser" (with appropriate sounds) at the front. This was kind of cool if you could get the dog to sit still long enough.

Reality Kind of Sucks
Programming this thing was frustrating (to say the least). There was no screen, which meant that you could never see the complete "program". This also meant that there was no way to fix a wrong step in the program. You had to start from the beginning each time. For complex routes, this was tedious.

Here's a sample that was in the instruction manual:

Sample Living Room

(The manual also had blank pages where you could draw in your own living room.) This sounds easy enough: Go forward 4, turn right 15, go forward 6, turn left 15, go forward 5, turn left 15, go forward 6, and then Fire!

The problem:

Physical reality doesn't match theoretical reality.

The tires on the Big Trak were sort of a rubbery plastic. This meant that the grip was unreliable on carpeting (or on tile). The program to move forward "5" may result in really moving "4-1/2". And turning "15" may result in turning "12".

Lather - Rinse - Repeat
To program this thing, I would always start with theoretical reality. I would map things out, write out the steps on a piece of paper, and then set out entering the program.

Then I would hit "Go".

And the Big Trak would promptly not turn quite far enough and end up running into the sofa.

Then I would adjust the first turn a bit, update my steps, and then enter the program again.

Then I would hit "Go".

And watch as the Big Trak makes the first turn okay, but doesn't go forward quite far enough and ends up crashing into the coffee table.

And this would continue. And after many iterations (and with a very mellow dog), you may be able to get it to drive around the coffee table, not hit the sofa, and then fire the laser at the dog on the other side of the room.

I Must be a Programmer
I really must be a programmer because this process did not frustrate me. It was a challenge. And there was something rewarding about getting the machine to finally do what you wanted it to do. It's hard not to get a bit nostalgic thinking about the afternoons I would spend "programming". (And if any of you share that nostalgia, Big Trak was resurrected a few years ago, and you can buy one on Amazon.)

I've always put the start of my programming experience in 1985 (when I was 14), but I really need to move it to 1979 (when I was 8). But I didn't actually get paid for programming until I was 29 -- which is a bit later than most professionals.

Debugging by trial-and-error is frustrating. Especially when you have to start over each time something goes wrong. I guess this isn't really that much different from people who entered computer programs by flipping switches on the front of a console.

Better Ways of Programming
It's much less frustrating working with tools today. I still do some programming by trial and error. For an example of this, you can look at "Coding Practice: Displaying Bitmaps from Pixel Data". But this is not my preferred way of working. Since I was using pixel data and creating bitmaps (which I didn't know in advance what they were supposed to look like), I really didn't have much choice. But this doesn't make up the bulk of what I do.

The bulk of what I do is implementing business rules, manipulating data, validating requests, and so forth. This is much closer to what I was doing last night: working on an implementation of Conway's Game of Life (more info if you're not familiar with it -- and something cool: if you Google "Conway's Game of Life", it shows a live demo of it).

I didn't get very far in my implementation. I started out by implementing the rules on whether a cell lives or dies. There are only 4 rules, so it's not very difficult. I used TDD for this -- creating tests for each rule and implementing them one at a time. This turned into more than 4 tests, though. I ended up with 14 tests to cover various states. And to simplify the tests, I'll probably move from MSTest to NUnit so that I can do parameterized tests for the various scenarios. (Why does MSTest only support parameterized tests for "Metro" applications?). This would reduce things down to 4 tests: (1) live cell stays live, (2) live cell dies, (3) dead cell come to life, (4) dead cell stays dead.

I'm still working on how to store the cell data. I'd like to do it in such a way that I can run the live/die method in parallel. I'm not quite sure how I'm going to do that at this point.

Wrap Up
Programming by trial-and-error is sometimes the only option -- like when programming a Big Trak. But more often than not, there are better options which are less frustrating and more productive.

It was eye-opening to realize that my first experience with programming had nothing to do with a "computer" (although there was a TMS1000 microcontroller inside). And it's also interesting to realize that I'm still programming by trial-and-error 35 years later.

But it's also good to know that this is not my preferred method of programming, and that I'm taking advantage of other options in the majority of my work.

I think that's enough nostalgia for now (or not, I may just have to order a Big Trak for myself).

Happy Coding!

October 2014 Speaking Engagements

I'll be traveling a little bit in October. If you're in Silicon Valley or the Phoenix area, be sure to stop by. These events should be a lot of fun.

Thursday, October 9, 2014
South Bay .NET User Group
Mountain View, CA
Meetup Event
o Abstract Art: Getting Things "Just Right"

Saturday & Sunday, October 11 & 12, 2014
Silicon Valley Code Camp
Los Altos Hills, CA
My Sessions
o Clean Code: Homicidal Maniacs Read Code, Too!
o Learn to Love Lambdas
o Community, Career, and You: A Microsoft MVP Panel

This will be my third time going to the Silicon Valley Code Camp. It is an amazing experience. Tons of sessions on all different topics to choose from (there are about 25 sessions to choose from in each time slot). I've met a lot of great people over the last two years, and I'm looking forward to being there again this year.

Saturday, October 18, 2014
Desert Code Camp
Chandler, AZ
Code Camp Site
o Abstract Art: Getting Things "Just Right"
o Dependency Injection: A Practical Introduction
o Learn to Love Lambdas

Desert Code Camp is one of my favorite events (I know, I'm not supposed to have favorites). This will be my 9th time at this event. The Phoenix area has a great developer community, and I always have a lot of fun when I go out there. I'm looking forward to talking to some old friends and also meet a lot of new people.

Abstract Art: Getting Things "Just Right" (the movie)
You may be wondering about my latest Pluralsight course. It's currently going through the final production processes and should be released the week of October 6th.

[Update 10/06/2014: It's published! http://www.pluralsight.com/courses/abstract-art-getting-things-just-right]

In the course, we look at how abstraction is awesome and how abstraction is awful. If we have too much abstraction, then our applications are complex and hard to maintain. If we have too little abstraction, then our applications are rigid and hard to maintain. Our goal is to figure out how to get just the right amount of abstraction in our applications.

We'll do this by looking at ourselves and how our own nature as developers can get in the way of achieving this goal. And the course is chock full of practical advice -- real practices that we can use to help us get things "just right".

And of course, you can also see the live version (which is a bit condensed) at upcoming events. Hope to see you in person soon!

Happy Coding!

No Estimates Minus Continuous Delivery Equals ???

I'm still thinking about No Estimates. So far, I've talked about how the business already behaves with a No Estimates mindset, about how developers need to be an integral part of the business, and about how organically grown software adds value to the business (and more here).

There are a lot of things that attract me to this mindset. And I also have quite a few reservations. I understand that tools have their pros and cons and that no tool is the right answer for every situation. The same is true of No Estimates. It definitely has a set of problems that it targets, but there are some other problems that aren't addressed. I don't expect that No Estimates has an answer to these problems, but I'm wondering if there are elements that we can incorporate into these other environments.

Continuous Delivery (or not)
Continuous Delivery is an amazing thing. If we can release updates to our software with a regular cadence at (hopefully) brief intervals, we can keep adding value to the business without having to wait for quarterly releases or "big bang" updates.

The examples of the organically grown software fit in nicely with the idea of continuous delivery. Although there wasn't a regular cadence, we did keep adding functionality in small pieces which gave the company value on an ongoing basis.

And for many of the other projects that I've worked on, many of them could have been split up into small chunks that were delivered on a regular basis (after hitting that "minimum viable product" stage). But there are some other projects that are more difficult to break down to fit into a continuous delivery process.

Quarterly Releases
As an example, I worked for a company that produced authorization software. Anything security related is already under high scrutiny. And on top of that, this was software that is installed at the client's site (and often isolated from external networks).

In this environment, there is no way that we could do weekly or monthly releases. When we had a new release, our clients would set up development and QA environments to test the new product. This included both functionality tests and penetration tests. Only after the software had gone through this process would they install it in the production environment. This process could take several weeks. So even if we were doing weekly releases, none of our clients would upgrade on that schedule.

So, you might suggest that we do the weekly releases, and then the clients can pick which features are important to them and upgrade when they find something compelling. And that initially sounds like a good idea. But our support system made this nonviable. When a client called with an issue, we would need to replicate their environment in the lab and then possibly provide a patch if the software had a problem.

This meant that we needed to keep working branches of prior releases available so that we could patch that particular release (without forcing the client to upgrade to the latest version).

"Big Bang" Release
As another example, at a large company, there was a project to replace an aging scheduling system. Scheduling tens of thousands of employees on a regular basis is difficult enough, but when you add the various rule sets that come from state regulation, company policy, and union contracts, things get complicated. (Fortunately, I was not on this particular project, but I did get to interact with a lot of people who were.)

This is where it becomes difficult to come up with the "minimum viable product" for the initial release. If you target one group of employees, you find out that their particular union covers more than just that group, so you need to expand the scope of employees included. If you manage to find an isolated group, you find that some of the employees move back and forth between departments (depending on how busy things are).

When you take the variables into account, the "minimum viable product" comes so close to "full replacement" that it doesn't make sense to do things incrementally.

And I'll be the first to say that this is a difficult way of doing things. There's no way to get everything right the first time (especially with this type of release), which means that the first six months after release are going to be a bit painful for the users and the development team.

Moving Into The Ridiculous
This can stray into the territory of "What the heck are you thinking?!?" I came across a group of business analysts who were collecting requirements for one of these large, company-wide projects. They were on a 24-month requirement gathering mission. Let me repeat that: 24-month requirement gathering mission.

That's 2 years! Even in a slow-moving corporation, things are going to change faster than that. By the time you get to the end of 2 years of requirements gathering, you'll just find that the business has changed, many of the features are no longer needed. And there will be a gap for features that are truly needed but came up too late to be included.

No Estimates - Continuous Delivery = ???
Now some of these examples are extreme, but it does bring up the question: can we get benefit from the principles behind No Estimates even if we are not in an environment where we can implement Continuous Delivery?

For example, in the quarterly release scenario, each of our clients had features that were important to them. It's not really possible to give each of them a "yes that's in this release" or "that will be in the next release" unless we are doing estimates on how long the various features will take, how much bandwidth we have from our development team, and what our planned release looks like. Definitely not a "No Estimates" mindset.

I don't have the answer to this. It may be that we need to come up with better ways of determining what the "minimum viable product" is. This is not too hard when working with green-field development. But when working on system-replacement projects, it's a bit harder to convince the business area that they need to stop using one function in the old system so that they can use it in the new system (especially when there is no practical way of actually disabling the feature in the old system).

It may be that we need to have better delivery mechanisms. If you are deploying just to your own servers, then it becomes much more practical to do weekly or daily releases. And there are some web-based companies that do multiple releases throughout each day.

This is a viable solution for server-based software that we run on our own servers or put into the cloud. It is more difficult if the software needs to be installed on servers at a client location. (And I would bet the majority of clients don't really like the idea of "auto-update" on their servers -- but that thinking will gradually change as people integrate more cloud solutions and see more and more successful updates.)

Wrap Up
As you can see, I've been thinking quite a bit about this. I really like the idea of No Estimates, development by subscription, and continuously adding value to the business. I think that I've identified a number of scenarios where this works well, and it really just requires buy-in from everyone involved.

But there are some scenarios where it doesn't seem to fit at all. I don't want to abandon the ideas (because they are good ones), and I'd like to see if there are small elements that we can bring in to these projects to improve the development process.

No tool is perfect for every job. But there's got to be some good principles that we can bring over into these other environments.

For more info, feel free to join the #NoEstimates discussion on Twitter, or look up Neil Killick or Woody Zuill. And I'll be glad to hear any of your insights on this topic.

Happy Coding!

More Organic Software: Escaping Into the Wild

A couple days ago, I wrote about how an application I wrote for myself escaped into the wild and became added value for the company. But I also had this happen with another piece of software that I wrote for myself. The way it grew organically based on business needs makes me think about how No Estimates can be appropriately applied in similar situations.

Working with Time Clocks
In addition to working with automated turnstiles, I also worked with the time clock system. Specifically, I worked with the time clock hardware and data collection. This was then transferred to the time tracking system (which was the "system of record" when it came to time tracking).

The time clocks worked with similar hardware as the turnstiles, and so I would check them every morning to make sure that the everything was online. Initially, I used a set of SQL scripts to check this, but as you can imagine, I ended up writing a little application to help me out with this.

This application had a drop-down box that listed all of the time clocks and a data grid to show the latest transactions. Again, I could easily scroll through the drop-down and see which time clocks were reporting data. This required a bit more thought, though -- some time clock locations were used more heavily than others. For example, the time clocks at the site entrances had a steady stream of transactions, but the time clocks that were in the maintenance shops were only used by a small group of people, so it was normal to have sporadic transactions.

Escaping into the Wild
The scheduling team would use the time tracking system throughout the day. They would make adjustments if someone clocked incorrectly (or forgot to clock in/out). And they would also check for "no shows" -- people who did not show up for a scheduled shift.

Because of this, the scheduling team was more dependent on having the data consistently sent to the time tracking system throughout the day.

One of the schedulers saw my application and said, "Can I have that?"

And the reason turned out to be pretty simple. If he saw a gap in the time tracking system, he could check to see if a particular time clock was off-line. If an employee regularly clocked in at the west entrance, he could check the 3 time clocks at that location to see if they were all reporting data.

If one of the time clocks appeared to be off-line, then the scheduler could contact my team to have it reset. This gave them better visibility to the system, and they knew that if a time clock was off-line, they would not necessarily mark "no shows" for people who normally clock at that location.

More Features
I think it's good when a business area asks for more features. It shows that they are thinking about the technology and data that we have provided to them and coming up with new ways to use it. So, I wasn't really surprised when I got a request for more features in the time clock viewer application.

We had security swipes at the entrances to the site. This would allow the security team to easily check that people were allowed to enter the property. The employees would swipe their card, and get a "go" or "no go" based on whether they were an active employee (cards would get automatically disabled if someone quit or was let go). But this also meant that people who needed to clock in had to swipe again at a time clock.

To streamline this process, we combined the security system and the time clocks. We used the time clock to collect the card swipe (either a bar code or RFID) and then pass the card information along to the security system for validation. This resulted in a single swipe for people who needed to clock in as well.

But there was a process issue: the employee would need to press the "Clock In" button the time clock before swiping his card. If you did not press the button, the security swipe would be processed and you would be allowed on property, but a time clock record would not be generated.

"I Forgot"
As you can imagine, when people do this everyday, sometimes they forget to push the button (as they are maybe rushing too quickly one day), or they think they pushed the button but it didn't register, or they simply forgot to check for the "clock in accepted" message that would normally display.

If someone forgot to clock in, the schedulers could manually add a record to the time tracking system. But because this was a common occurrence (especially when there are several thousand people clocking in throughout any particular day), the schedulers wanted a bit more information.

You see, some employees had figured out a way to game the system. If they got to the clocking location and found that they were 2 minutes late, they could "forget" to press the "Clock In" button and ask their scheduler to override it. This would mean they wouldn't get penalized for being late.

So the schedulers asked if there was a way for them to get the time clock data combined with the data from the security system. This way, they could look up an individual and see what time the security swipe went through. This would let them check to see if people were actually late when they came in, and it would allow them to separate the people who forgot to clock in from the people who "forgot" to clock in.

Business Changes - Things Get Complicated
Whenever you deal with time tracking, things get really complicated. There are federal and state regulations, there are company policies, and there are union rules -- all of which need to be followed.

One change that came about was the ability to clock in and out at your work location (instead of the property entrance). The reason is that it could take 10 to 15 minutes to get to your actual work location from the property entrance. You were supposed to be at your location and ready to work at your scheduled time, so having a time clock at the location would help better track that.

Unfortunately, the time tracking system didn't have this level of granularity. It kept track of "clock in" and "clock out", but not the particular time clock that was used.

So, we were asked to come up with some printable reports that could be used by the location managers based on the raw time clock data. These reports would allow a manager to see all of the clock transactions for employees in their area (based on their department identifier). The manager could double-check that their employees were clocking at the right locations, and they could coach employees who needed a reminder on procedures.

Later on, we extended the functionality of these reports to show "exceptions" -- such as people who clocked in but forgot to clock out, or for people who clocked in and out at different locations. This let the management teams focus on where specific problems lie.

And all of this helped the company comply with the various contractual rules and legal regulations that governed the time tracking of employees.

Wrap Up
So, here's another example of an application that grew organically as the business areas saw places where we could add value. And it's another good example of how No Estimates could work well in a changing environment.

And even though I've come up with some good examples of where No Estimates would be effective in projects I've worked on, I've also been thinking about some situations where it would not be a good choice. (I'll be writing about these soon.)

This doesn't bother me, though. No tool is perfect for every situation. I wouldn't expect that No Estimates would be a good choice for every situation, either. What's important is that we take a look at our current processes, determine what's working and what's not working, and look toward improving things where we can.

Happy Coding!

Observation: A Key Debugging Tool

When I was writing about attraction turnstiles yesterday, one of my favorite troubleshooting stories came to mind. And this is one situation where the data and system logs didn't give us any insight. It was only when I went out to observe with my own eyes that the problem came to light.

Different Technologies

As I mentioned last time, I worked at a major theme park and was part of the project to convert from manual turnstiles to automated turnstiles (more of the story). As another part of the project, we were moving away from mechanical turnstiles (the familiar 3-bar clicky things like in the picture) to optical turnstiles. Optical turnstiles use infrared light beams to count people walking by. These are usually invisible to the people walking by (unless you're someone like me who specifically looks for them).

Optical turnstiles are a bit less accurate than the mechanical ones primarily because they don't act as a barrier, and so people walk through them differently. As expected, there was a concern by the operating area that the optical turnstiles would not be accurate enough for their needs. But after much research, implementation tests, and on-site trials, it was determined that the accuracy was adequate for the business needs. And of course, one of the big benefits was that they were invisible and provided a better experience for our customers.

Data Discrepancies
At most locations, the optical turnstiles performed as expected. But at one location, they had regular inconsistencies. This particular location happened to be a playground for kids -- lots of cargo nets to climb, caves, towers, bridges, rock climbing walls. And the entrance and exit were combined.

Because of the nature of the location, the turnstiles counted in both directions, meaning they counted the people coming in as well as the people going out. This allowed us to keep a "running total" of how many people were in the location at any one time.

But there was a problem. This running total never went down to zero. It would gradually creep higher and higher during the day. At the end of the day, it would show several hundred people who went in but never came out. Now this obviously wasn't the case (unless there was a portal to another dimension somewhere in the caves -- but we didn't have any complaints about missing persons (no, not that Missing Persons (I still miss the 80s))).

Initial Troubleshooting
This sounded like a technical problem, so we got right on it. I checked the raw data from the location for a several week period. There were 4 different "lanes" (separate turnstiles). I looked for gaps in the data that would indicate that a sensor went offline. I looked to make sure we got counts in both directions (both entry and exit counts) from all of the lanes.

There wasn't anything obvious in the data, so it was time to move on to Stage 2: verifying the counts manually.

Observation

I grabbed by trusty person counter (like the one pictured), synchronized my watch with the server, got my clipboard, and headed out to the location.

I would not have been surprised to find a problem with the optical sensors. Since the sensors worked on infrared light, they could be "blinded" by the sun. If they were hit by direct sunlight, they wouldn't be able to make any counts at all. Since this was an outdoor location, I expected I might see some of that.

I was also there to observe behavior of both the customers and employees. If it turned out that an employee would stand in front of one of the sensors, it could affect the counts. We had seen locations were ropes were put up that blocked the sensors or swinging flags would cause the sensors to count even when no people were passing through.

Since there were 4 lanes at the location, I prepared myself to be there for a while. I need to count at least one 15-minute interval for each lane, and I also wanted to count the entry and exit separately. When my watch indicated the beginning of an interval, I started manually clicking off the people using the first lane to enter.

Realization
It was only by standing out there for 2 hours that my brain started to register what the real problem was. This was a kid's play area. It was a place where parents would take their children to expend their excess energy. Kids were climbing over everything, running (even though it wasn't allowed), jumping, swinging, and generally tiring themselves out.

And what happens when kids get tired? They want their parents to carry them. And that's exactly what was happening. Children were walking in to the location (triggering an entry count) and being carried out (*not* triggering an exit count).

While I was watching this (and still clicking away), I did some quick calculations in my head:

Assuming 50% children and 50% adults, if 10% of the children walked in and were carried out, it would account for the discrepancy that we were seeing in the numbers.

So, I had the answer even before verifying my manual numbers. I completed my counts and went back to my desk. And I found exactly what I expected to find: the manual counts that I took matched the automated counts on the server.

I was only able to identify the problem by making observations with my own eyes. I didn't find what I was looking for. But by being at the location and seeing the normal behavior, I was able to figure out what was happening.

Wrap Up
Not all problems with our systems are technical. Sometimes there are human elements involved. Only by watching how people actually use the system can we determine these types of issues. Sometimes it's an easy fix. If I see that one of my users wants to click on "Step 2" before completing "Step 1", then that tells me I need to hide some things until Step 1 is completed -- guiding the user to success in every part of the application.

In the case of the turnstile discrepancy, there wasn't much we could do to alter the customer behavior. We did add a "Minus 1" button at the location. If the employee noticed kids being carried out, she could press the button to trigger an exit count. This wasn't a great solution since the employees are usually doing other things like talking to the customers and answering questions. Other technical solutions would be overly complex and expensive for this particular implementation.

But having an answer to the question was really key in this case -- this made the operations folks confident that the optical sensors were dependable from a technical perspective, and they could rely on the system as a whole.

We can't always find problems from looking in logs and checking data. Sometimes we need to go out and see things with our own eyes.

Happy Coding!

Organic Software: Spreading Usefulness

I've often mentioned that I've built the most successful software by understanding the users and their day-to-day business. But sometimes, I make something that is useful for myself, and it grows in unexpected ways.

Working with Turnstile Counts
I spent may years working for a major theme park. I actually worked at the company for 5 years before joining the I.T. team as a junior developer and hardware assistant. As an assistant, I worked with Doug (the technical lead) on automating the attraction turnstiles. Prior to automation, turnstile counts were collected on the device (usually a mechanical turnstile with an LCD screen), and the employees working at the location were responsible for writing down the counts from the LCD screen every hour. Then they would manually enter the counts into a tracking system.

The automated turnstiles were network-connected and sent their counts to a server every 15 minutes. This was great for the people working at the location because they no longer had to do the manual recording process. On top of that, we could automatically transfer the counts to the tracking system.

But there was a problem: the early versions of the network modules were a little flaky and would go off-line. None of the data would be lost, but the network module would need to be reset to send the data to the server.

One of my jobs was to check the turnstile controllers every morning. This involved running some SQL scripts against the database and looking for "gaps" in the data. If a particular controller had not reported data for a while, I knew that the controller needed to be reset.

Lazy Programmer - Productive Developer
As we converted more and more attractions, this process became a bit more tedious. When there are just a few controllers to look for, the gaps were very obvious. But when there are several dozen controllers, the gaps in the data are harder to find.

I'm a lazy programmer, and so I did what any lazy programmer would do when faced with a repetitive task: automate it.

Okay, so I didn't go full automation. But what I did was put together a simple application. It just had a drop-down box and a data grid.

The drop-down box had a list of all of the turnstile controllers. When a controller was selected, the grid would show the data for that one controller (with the most recent record on top). This made the job of checking the controllers very easy: I could just scroll through the drop-down box and look to see which items did not have current data.

The Application Escapes
One day I was reviewing the data with one of the managers from the area. When she saw my little helper application, she said, "Hey, what's that?"

I demoed the application and showed the raw data that came from the turnstile controllers. And she said, "Can I get that?"

The reason for this was simple: the manager had access to the tracking system which gave data that was one day old (yesterday's data). The little application that I had provided current data (well from the latest 15 minute data block, anyway). The manager really liked the idea of being able to access the data throughout the day. This way, it could be acted upon immediately.

For example, if the counts were zero, that may indicate that that the attraction was down (maybe due to a mechanical fault). If the counts were lower than expected, it could indicate that there is a staffing problem at the location that lowered the throughput. For either of these situations, the manager may want to go to the location to check what's going on.

So, I cleaned up the application a bit. I parsed the raw data into columns that the operations folks could easily understand. And I also added some total columns (for example, if there were 3 turnstiles at a location, they came through as separate records, but the operations folks were more interested in the total counts for the location. Another example, the counts came in 15 minute intervals, but they wanted to see hourly totals since that's what their operational metrics were based on).

Soon, there was an application that I had originally built for myself that was now being used by about 50 people to help them do their jobs better.

More Functionality
After the conversion from manual turnstiles to automated turnstiles was complete, this application now had all of the raw data for the locations. The managers and leads could check the data throughout the day.

As expected, the operations folks came up with some new ideas for the data. First was the idea of notifications.

At regular intervals throughout the day, they wanted to receive email or pages that had the current data. This meant that people working in the area (and not at a computer) could still take advantage of the data.

This wasn't difficult for us. We already had a notification system that could send out email or pages to different groups and individuals (we had lots of Blackberrys and Motorola pagers in use at the time). In fact, the biggest part of adding this task was to build some maintenance screens. Each person could choose which data they wanted (they could select just a few attractions or all of them) as well as the times they wanted to be notified (for example, every hour during normal operating hours) as well as the days of the week to be notified (so they would only get the information on days they were working).

This added about 50 more people who could access the data on a regular basis to help them keep the business functioning efficiently.

Yet Another Feature
The next step was to add "alerts". For each location, they could enter a target number and a warning threshold. For each page, we would include the current count plus the target number for reference. And if the current count fell below the warning threshold, that would trigger a separate set of "alert" pages.

And Another
The last major feature that we added was a summary email that was sent out every morning. This would include a summary of the prior day's data so that directors and area managers could track how things were going. This replaced a manual process that the attractions staff was doing.

Now the leadership team was able to directly benefit from the data collected.

Organically Grown
Now this isn't a process that happened overnight. These features grew over a period of several years. But the interesting thing about this is that it was not a set of features that was originally requested by the business area. It was something that I built for my own use that happened to have usefulness beyond just me.

And I think that's why I have such an affection for this particular application. This wasn't a "project" that the business area came up with. When the turnstiles were being automated, the primary benefit was to eliminate the labor spent collecting the counts manually (and the potential for transcription error) and putting the data into the tracking system (with potential for more errors).

No one asked for a system that would page (near) real-time numbers to the operating areas. But once we had the data, and the users in the business area started to see the how they could benefit from it, things just grew naturally.

We discovered value incrementally over time. And the end result had a lot more benefit than anyone had anticipated.

Wrap Up
So, this brings us back to the idea of "No Estimates". It's not just that we are bad at estimating features that are asked for by the business area. It's that we don't even know where the real value lies when we start out on a project.

This really shouldn't be a surprise. One of the major tenets of Agile is that we adjust our direction when we need to. When we see potential value, we're allowed to shift our current direction to pursue that. And these feature requests came in small chunks. At the beginning, I never would have been able to put together a viable estimate/plan for "send out current turnstile counts to selected individuals". But because we took things one step at a time (raw data, formatted data, summarized data, notifications), we were able to get to this unexpected result that added value to the company.

I guess I'm convincing myself more and more that I like the idea of No Estimates. When I look at projects that added real value to the company (like this one), it seems like we pretty much worked that way. (There was some project overhead, but minimal compared to most of the applications we worked on.)

I've got a couple more stories like this one that I'll be sharing soon. In the meantime, keep thinking about ways that you can be an effective developer and add the most value to the company.

Happy Coding!

A New Milestone

Thanks to you (and other folks who come to read my articles), this blog hit a new milestone: 100,000 page views!

This is an all-time view count that goes back to the first article published in 2009, so it took about 5-1/2 years to get to this point. It's not very impressive compared to popular blogs (some of my friends get as many page views in a day as I get in an entire month). But it's pretty good for someone who isn't famous (yet).

Just 2 years ago, there were about 600 views per month, and now it's over 5,000. A lot of that comes from persistence: over 200 articles and many that were written several years back still get regular views.

Thanks for reading. I hope that the articles have been helpful. There will be many more to come.

Happy Coding!

I Am Not A Supplier

Based on yesterday's thoughts on No Estimates, the discussion rolls back around to how I see myself as a developer:

I am not a supplier.
I do not simply provide services.
I am part of the business.
I provide value that moves the business forward.

As a developer, my goals are the same as the business I work for. I want to see the company succeed. The way I do that is to make the business run easier, more smoothly, and economically through the software that I build.

I am just as much a part of the business as Marketing, Strategic Planning, Operations, Sales, HR, and Maintenance. My role as a developer moves the company forward just like every employee of the company.

But this is not necessarily how the company sees me.

Just a Supplier?
This is a topic that I've written about before, and my thoughts have been coalescing over the years. At a company that I worked for, there was a Vice President of I.T. who had a goal: he wanted to be asked about the color of the carpeting in the new building.

What does this have to do with I.T.? Absolutely nothing. And that's exactly the point. In the executive meetings, the VP of Marketing was asked these types of questions, even though it has nothing to do with marketing. The VP of HR was asked these types of questions, even though it has nothing to do with HR. These groups were seen as integral parts of the company.

The VP of I.T., however, was only asked questions that had to do with technology. And this is indicative of the overall issue: I.T. as a supplier rather than a part of the company.

Coding to Specification Only

Unfortunately, many software developers see themselves as suppliers. They are not interested in understanding the business but are perfectly happy to simply code to a specification that someone else gave them. I've never been successful working that way. Only by being part of the business (and understanding the goals and how it works on a day-to-day basis) have I provided significant value as a software developer.

Common Goals & Vested Interest
This attitude of "just a supplier" has led to some issues that many companies are just now realizing. And it really comes to light with the phenomenon of outsourcing.

When you think of I.T. as a supplier, then it makes perfect sense to outsource it. It's not part of our core business, so we can get it from somewhere else -- like a utility. But then the reality of the decisions set in.

When a group is outsourced, they no longer share the goals of your company. Let's take a look at a scenario: your company sells consumer products on-line. The primary sales time is Christmas season (with a huge influx of sales on Cyber Monday).

Scenario 1: Common Goals
When the I.T. department is part of the company, then there is a common goal: success of on-line sales. So if credit card processing goes down on Cyber Monday, it's an "all hands on deck" situation where everyone is troubleshooting the issue. The network team is checking connections to the card processor; the server team is making sure that that servers are not overloaded; the database team is making sure that transactions are being committed; and the development team is checking to see how the applications are handling the loads.

Everyone is working toward one thing: getting on-line sales back up. It's what we do. It's the whole purpose for being there.

Scenario 2: Disparate Goals
Let's run that situation again, but with an outsourced I.T. department. Now the goals are different. The goal of the company is still the success of on-line sales, but the goal of the outsource company is to fulfill the service level agreements (SLAs) that are in the contract. The success of the outsource company is not dependent on the success of your company.

So if credit card processing goes down, there are trouble tickets that are opened. The company is concerned about every minute of outage due to lost sales. But the outsource company is concerned about making sure they fulfill their contractual obligations. This probably does *not* result in an "all hands on deck" situation (because that would have made the cost of the contract too high to be practical).

Unfortunately, I have seen this happen. Before outsourcing, we had dozens of team members troubleshooting various issues. After outsourcing, there was a conference call with the account managers and very little actually gets done.

There Is No "Us" and "Them"
One reason that the company treats I.T. as a supplier is because that's how we often act. We don't even act like a cohesive team within I.T. For some reason, we (as technologists) don't like to be seen as the party at fault. It may just be a side-effect of the other traits that make us good at working with technology. We like to be seen as infallible problem-solvers.

This leads to an "us" and "them" attitude with other teams inside I.T. So when a user calls in an issue with an application, we might say, "The application is working fine, so it's not us. It might be the network or database that's having problems. I'll forward you to the network team."

But we need to look at things from the user's perspective. The user doesn't care what the exact problem is or who is responsible for it. All he knows is that the application is not working, and he cannot do his job.

We need to react to the users with this in mind. If someone calls in an issue with an application, we should say, "I can't find the problem in the parts that I can see. Let me work with some other members of our team to figure out what's going on." Then work with the networking or database teams to try to nail down the issue.

And after solving the issue, the proper response to the user is, "We fixed the issue, and we're working to make sure that it doesn't happen again. But be sure to let us know if you run into the same thing." The user does not need to know whether the problem was the application, the database, or the network. The user doesn't care. All he wants to hear is that "we" (the I.T. team) have things under control.

Forming a Partnership
I've written many times about forming a partnership with the business areas. And I have been most successful as a developer when I had a good understanding of what my users needed to do on a daily basis -- the steps they went through to do their jobs. When I understood this, I could automate the little things to make their jobs easier. And I could make suggestions on bigger things that could streamline processes.

I knew that I was successful at building these partnerships when our company was hit by a round of layoffs. Most departments took a 10% staff cut. The first thing that happened was that my users were calling me for help, "We just lost 40 hours of labor; can you help us automate this process that takes 8 hours a week? That will take off some of the strain as we re-distribute work and re-prioritize."

It was these partnerships that kept us all moving forward in the same direction.

Wrap Up
What does this have to do with No Estimates? It's really a first step. Last time, we talked about the trust that is required between the business and development teams. But what if we eliminate that line that divides these two concepts? What if instead of having "the business" and "the developers", it was all just "us" -- a group of people all working toward the same goal.

I don't think of myself as a supplier. I am part of the business. I add business value. And when the business moves forward, I move forward with it.

There are times that I wished I ruled the world. I would try to spread this attitude to all of the developers who think of themselves as suppliers. And I would try to spread this attitude to all of the companies that think of their I.T. departments as suppliers.

In this world, everyone would be working toward the same goals. I.T. teams would be a true part of the business and continually add value. And companies would be more successful.

These changes are made one person at a time. I hope that you join me in this quest.

Happy Coding!

No Estimates: Development by Subscription

Last week, I spent a couple of days at Agile Open SoCal. It was a great time to talk about various issues and successes and get some great advice from other people who are in similar situations. There's a lot for me to process, and a couple of topics really stuck out.

One of these was No Estimates. I'll have to admit that I've been following this discussion from the sidelines. If you want to catch up on what No Estimates is, check out what Neil Killick and Woody Zuill have to say. I will not do the description justice, so you'll want to go to these guys who have been thinking about it for a long time.

I know Woody from the SoCal developer community, and I had a chance to sit down and talk to him a bit last week. (He's also known for showing the success his team has had with Mob Programming).

We Suck at Estimation
One of the basic truths is that we (as developers) suck at estimation. But that's not really a surprise. We are constantly being asked to do things that we've never done before.

An experienced cook will be able to tell you how long it will take a bake a cake. After all, he's done it a dozen times and knows what to expect. But for someone who's never baked a cake (like me), I would end up "padding the estimate" on how long it will take me. Is it because I don't know how to follow a recipe? No. It's because I'm working with unfamiliar tools -- I don't know exactly how to use the mixer or if I'm going to mess up a batch by over-mixing or under-mixing.

And this is true in software. Whenever I'm looking at a new project, it usually contains 4 things I've done before and 1 thing that's new to me. Now, I'm not completely ignorant of the technology (I've seen someone use the mixer), but I don't know what problems to expect (if you don't grease the bowl before you start, you won't get an even mixture).

I do not like to not deliver on my commitments -- meaning, I would rather over-estimate and deliver early than under-estimate and deliver late. The same is true in the rest of my life: I'd rather arrive at an event an hour early than 5 minutes late.

This means that I have a tendency to over-estimate:



I don't do this so that people will think of me as a miracle worker. I do this because I want to make sure I deliver on my commitments.

But this doesn't make my estimates any better than someone who under-estimates. In both cases, our estimates are wrong.

Eliminating Estimates
So, the idea of No Estimates is that we get rid of the estimation process. Instead of coming up with a huge project plan with lots of incorrect estimates (that end up compounding the problem), we focus on continually delivering small pieces of value.

I like the idea of this. And as a responsible developer, I think that it would work well for me and my development style -- I like to make my users happy, and I can best do that by constantly giving them features that make their jobs easier.

But I have been skeptical for a number of reasons. First, not all developers can work this way. There are some developers who would take advantage of the situation and slow down their delivery. There's no longer things like "story points" and "velocity" that measure productivity. And even though those measurements are probably wrong, people like to have something tangible to look at. The solution to this is to have managers who can keep track of productivity (in other ways) and know the capabilities and output of their teams.

Second, many companies cannot work this way due to their budgeting process. The financial systems are built around large projects and schedules. Marketing wants this and Sales wants that and Operations needs this other thing. So let's figure out how long each thing will take, how much our development team can handle at one time, and then prioritize everything. On top of that, money for the projects comes out of different buckets depending on whether it is a capital or expense project. I have no idea how to resolve this issue.

No Estimates is more of a subscription model for development. Instead of the business areas putting aside money for Project X that will cost $50,000 (maybe), the development team is funded on an on-going basis -- just like the other teams (finance, sales, operations, maintenance) that add value to the company.

The Business Areas Already Do This
I had a realization about this based on my experience: even though the company does not work this way, the business areas do.

I worked at a large company (20,000 people on site, 100,000 people in the division), and I built software for the various business areas based on what came down from project planning. And this was a hassle for many of the business areas. Sometimes a department just needed a small piece of functionality. It wasn't large enough to justify an entire project -- maybe just a report to collate data. And even if it were big enough, it probably would not be prioritized near the top of the list. So it may take a year or more to get it on the schedule.

The solution: Shadow IT. I don't know if there's an industry name for this phenomenon, but this is what we called it in our company. A department would have someone on staff who was more technical than most and probably good at something like Microsoft Access. This person would then build all of these small pieces of functionality that the department wanted to make their job easier.

So the business areas have already bought into the idea of development by subscription. They have decided that it was worthwhile to pay someone in their department to handle these types of things. And this Shadow IT person was paid out of the same budget as all of the other employees in the department.

(And this arrangement isn't all puppies and rainbows. Since the majority of the Shadow IT people did not have any formal technical training or experience, some of the projects would run into scaling issues or simply stop working when the data got too large. At this point, the systems were critical to the department, so they would often turn to the development team to fix the issues or rewrite the system.)

So the business areas have recognized this as a way to add value to the company. Now we just need to get the company itself to realize it.

This would give the opportunity for the business areas to have access to actual developers who can deliver functionality quickly in a supportable way.

Wrap Up
I really like the idea of No Estimates, and I hope that it gains traction. There is a lot of trust required between the company and the development teams (but I've wanted to have better integration between the company and the development teams for a long time -- that trust is needed anyway).

It was an eye-opener when I realized that the business areas had already implemented their own version of No Estimates by having a Shadow IT person on staff. Next, we need to figure out how to formalize this so that the right people (developers) are doing the right work.

Happy Coding!