Are You an Over-Abstractor or an Under-Abstractor?

I really like to teach people about different techniques of abstraction. If you don't believe me, just check my website: Interfaces, Dependency Injection, Design Patterns, and even the BackgroundWorker Component (and also my courses on Pluralsight).

One of the reasons that I talk about these particular topics is that they are things that took me a really long time to understand myself, so I like to help people take a short-cut to understanding. But that leads to a problem. Once we learn about them, we might want to use them everywhere, and we end up building a complex mess that's hard to maintain.

But over-abstraction is not our only problem. If we have too little abstraction, then our applications can be rigid and hard to maintain. So as developers, our goal is to find the sweet spot right in the middle. This is why I came up with "Abstract Art: Getting Things 'Just Right'" (which I give as a live presentation and as a course on Pluralsight). We're really looking to follow the Goldilocks Principle when it comes to abstraction: not too much, not too little, but "just right".

The Goldilocks Principle (from the presentation slides)

Observing Developers
So why do we, as developers, have a problem getting abstraction right? There are a number of factors, but the biggest one has to do with our natural tendencies. I'm not sure if anyone has done an official study on this, but I formed this opinion based on my observations of developers that I've worked with (and also by reflecting on myself).

The premise is that every developer has a default state as either an over-abstractor or an under-abstractor. Over-abstractors like to build stuff and think about all the possible uses of an application in the future. This may lead to overly complex implementations for features that will probably never get used. Under-abstractors like to keep things simple. This may lead to things like global variables, tightly-coupled components, and "god classes" with all sorts of functionality.

This is just our "default state" -- meaning, without any external influences, we lean one way or the other. But as we gain experience as developers, we run into difficulties with our default state and so we go in the other direction. This often results in us swinging too far, and we end up correcting back the other way.

The Pendulum Effect (from the Pluralsight course)

Eventually, we figure out the dangers of both over-abstraction and under-abstraction through trial-and-error. The main reason I wrote my presentation and Pluralsight course is to give people some best practices so they can skip some of these bad experiences.

One Group is Not Better Than the Other
The first time I gave this presentation, I ran into a bit of a problem: one of the attendees thought that I was promoting one group over the other -- for example saying that under-abstraction is better than over-abstraction. And that is definitely *not* the case.

Both groups have problems. With too much abstraction, we end up with code that is overly-complex, difficult to debug, and can cause frustration in people who are not intimately familiar with the code (which may include the original developer 6 months later). Ultimately, the code is difficult to maintain.

With too little abstraction, we end up with code that is rigid. It is hard to find bugs and very difficult to test. Ultimately, the code is difficult to maintain.

So we have a common result: code that is difficult to maintain. What we need to do is figure out how to recognize our natural tendency so that we can watch for the warning signs and hopefully end up with abstractions that are "just right" for our application.

Do You Meet More Over-Abstractors or Under-Abstractors?
In my presentations, I would love to do an informal survey to see if the developers identify themselves as over-abstractors or under-abstractors. But because I want to be careful not to create "teams" in the room, I don't do that.

I've been curious if there are more over-abstractors or under-abstractors in our field (or if there's an even split). I've been able to make the case of both groups.

The Case for Over-Abstractors
The main reason I would think that there are more over-abstractors in our field is that what we do is all about problem solving. We are developers because we like to come up with creative solutions and then implement those solutions by building things.

And a big part of that is the building. We don't want to use code that someone else wrote; we want to write it ourselves. This is often referred to as the "not invented here" syndrome.

Another part of this is that we like to plan for the future. Sure, we could build a robot that fetches a soda from the refrigerator. But wouldn't it be cooler if the robot would also vacuum as it goes? (Yeah, I know, we already have a vacuum cleaner that works well, but it will still be cool.) This is a fairly common mindset that we see in developers -- we are creative people.

Based on this, I might think that there are more over-abstractors in our industry.

The Case for Under-Abstractors
The main reason that I  would think there are more under-abstractors in our field is that there are a lot of developers who do not understand abstraction or what it is good for. As a young developer, I had a lot of trouble understanding why I would want to use interfaces in my code. I could tell they were important, but I couldn't find the right person to explain it to me. (This is one of the main reasons that I like to talk about Interfaces -- I don't want developers to go through the same struggle that I did.)

I come across a lot of developers who are like I was: they do not understand the basics of abstraction. I meet some of these developers when I'm helping them with their code; I meet some of these developers at my presentations. And my goal is to give them a better understanding of the topic while I have a chance to talk to them.

Based on this, I might think that there are more under-abstractors in our industry.

Any Ideas?
So, what are your thoughts? How do you identify yourself? Do you have more over-abstractors or under-abstractors where you work? What about the developers you meet at conferences or other events?

If you're not sure where you fall, you can check out the "Default State Quiz" that's part of my Pluralsight course.

By nature, I am an under-abstractor, but I've also swung into the over-abstraction world for a while. After my many years of experience, I like to think that I'm pretty well balanced, but I still have a tendency to lean toward my "default state" from time to time.

The first step to overcoming my default state was to recognize it. Since I've done that, I can now catch the times when I'm leaning too far toward under-abstraction. This gives me a chance to review some best practices to steer myself to a more balanced state.

Feel free to leave a comment with your thoughts (or drop me a note privately). I'm interested to hear what you think about the topic.

Happy Coding!

Programming by Trial-and-Error

I guess I must be getting old because I keep looking back at my history as a programmer to see what I can learn about myself and what works and what doesn't work. This week's realization is that I've been a programmer for a lot longer than I thought.

When I've written about my history, I usually mention that I wrote my first program on a knockoff Apple ][ way back in 1985. But that's not accurate. I wrote my first program long before that.

Christmas 1979
Way back in 1979, I got an awesome toy for Christmas: Big Trak. This was not a remote-control vehicle; it was a programmable one.

Merry Christmas!

It had a keypad on the back that you would use to punch in instructions (up to 16 of them). Then you press "Go", and it follows them (sort of) exactly.

It could move forward or backward in the very precise unit of "vehicle length".

It also could turn left and right. The units for this were a little easier: "15" was a 90 degree turn. It doesn't sound very intuitive, but there were 60 units in a full circle, which meant that it was similar to a clock face (we used analog clocks back then).

Finally, there was a "Fire" button that would fire off the blue "laser" (with appropriate sounds) at the front. This was kind of cool if you could get the dog to sit still long enough.

Reality Kind of Sucks
Programming this thing was frustrating (to say the least). There was no screen, which meant that you could never see the complete "program". This also meant that there was no way to fix a wrong step in the program. You had to start from the beginning each time. For complex routes, this was tedious.

Here's a sample that was in the instruction manual:

Sample Living Room

(The manual also had blank pages where you could draw in your own living room.) This sounds easy enough: Go forward 4, turn right 15, go forward 6, turn left 15, go forward 5, turn left 15, go forward 6, and then Fire!

The problem:

Physical reality doesn't match theoretical reality.

The tires on the Big Trak were sort of a rubbery plastic. This meant that the grip was unreliable on carpeting (or on tile). The program to move forward "5" may result in really moving "4-1/2". And turning "15" may result in turning "12".

Lather - Rinse - Repeat
To program this thing, I would always start with theoretical reality. I would map things out, write out the steps on a piece of paper, and then set out entering the program.

Then I would hit "Go".

And the Big Trak would promptly not turn quite far enough and end up running into the sofa.

Then I would adjust the first turn a bit, update my steps, and then enter the program again.

Then I would hit "Go".

And watch as the Big Trak makes the first turn okay, but doesn't go forward quite far enough and ends up crashing into the coffee table.

And this would continue. And after many iterations (and with a very mellow dog), you may be able to get it to drive around the coffee table, not hit the sofa, and then fire the laser at the dog on the other side of the room.

I Must be a Programmer
I really must be a programmer because this process did not frustrate me. It was a challenge. And there was something rewarding about getting the machine to finally do what you wanted it to do. It's hard not to get a bit nostalgic thinking about the afternoons I would spend "programming". (And if any of you share that nostalgia, Big Trak was resurrected a few years ago, and you can buy one on Amazon.)

I've always put the start of my programming experience in 1985 (when I was 14), but I really need to move it to 1979 (when I was 8). But I didn't actually get paid for programming until I was 29 -- which is a bit later than most professionals.

Debugging by trial-and-error is frustrating. Especially when you have to start over each time something goes wrong. I guess this isn't really that much different from people who entered computer programs by flipping switches on the front of a console.

Better Ways of Programming
It's much less frustrating working with tools today. I still do some programming by trial and error. For an example of this, you can look at "Coding Practice: Displaying Bitmaps from Pixel Data". But this is not my preferred way of working. Since I was using pixel data and creating bitmaps (which I didn't know in advance what they were supposed to look like), I really didn't have much choice. But this doesn't make up the bulk of what I do.

The bulk of what I do is implementing business rules, manipulating data, validating requests, and so forth. This is much closer to what I was doing last night: working on an implementation of Conway's Game of Life (more info if you're not familiar with it -- and something cool: if you Google "Conway's Game of Life", it shows a live demo of it).

I didn't get very far in my implementation. I started out by implementing the rules on whether a cell lives or dies. There are only 4 rules, so it's not very difficult. I used TDD for this -- creating tests for each rule and implementing them one at a time. This turned into more than 4 tests, though. I ended up with 14 tests to cover various states. And to simplify the tests, I'll probably move from MSTest to NUnit so that I can do parameterized tests for the various scenarios. (Why does MSTest only support parameterized tests for "Metro" applications?). This would reduce things down to 4 tests: (1) live cell stays live, (2) live cell dies, (3) dead cell come to life, (4) dead cell stays dead.

I'm still working on how to store the cell data. I'd like to do it in such a way that I can run the live/die method in parallel. I'm not quite sure how I'm going to do that at this point.

Wrap Up
Programming by trial-and-error is sometimes the only option -- like when programming a Big Trak. But more often than not, there are better options which are less frustrating and more productive.

It was eye-opening to realize that my first experience with programming had nothing to do with a "computer" (although there was a TMS1000 microcontroller inside). And it's also interesting to realize that I'm still programming by trial-and-error 35 years later.

But it's also good to know that this is not my preferred method of programming, and that I'm taking advantage of other options in the majority of my work.

I think that's enough nostalgia for now (or not, I may just have to order a Big Trak for myself).

Happy Coding!

No Estimates Minus Continuous Delivery Equals ???

I'm still thinking about No Estimates. So far, I've talked about how the business already behaves with a No Estimates mindset, about how developers need to be an integral part of the business, and about how organically grown software adds value to the business (and more here).

There are a lot of things that attract me to this mindset. And I also have quite a few reservations. I understand that tools have their pros and cons and that no tool is the right answer for every situation. The same is true of No Estimates. It definitely has a set of problems that it targets, but there are some other problems that aren't addressed. I don't expect that No Estimates has an answer to these problems, but I'm wondering if there are elements that we can incorporate into these other environments.

Continuous Delivery (or not)
Continuous Delivery is an amazing thing. If we can release updates to our software with a regular cadence at (hopefully) brief intervals, we can keep adding value to the business without having to wait for quarterly releases or "big bang" updates.

The examples of the organically grown software fit in nicely with the idea of continuous delivery. Although there wasn't a regular cadence, we did keep adding functionality in small pieces which gave the company value on an ongoing basis.

And for many of the other projects that I've worked on, many of them could have been split up into small chunks that were delivered on a regular basis (after hitting that "minimum viable product" stage). But there are some other projects that are more difficult to break down to fit into a continuous delivery process.

Quarterly Releases
As an example, I worked for a company that produced authorization software. Anything security related is already under high scrutiny. And on top of that, this was software that is installed at the client's site (and often isolated from external networks).

In this environment, there is no way that we could do weekly or monthly releases. When we had a new release, our clients would set up development and QA environments to test the new product. This included both functionality tests and penetration tests. Only after the software had gone through this process would they install it in the production environment. This process could take several weeks. So even if we were doing weekly releases, none of our clients would upgrade on that schedule.

So, you might suggest that we do the weekly releases, and then the clients can pick which features are important to them and upgrade when they find something compelling. And that initially sounds like a good idea. But our support system made this nonviable. When a client called with an issue, we would need to replicate their environment in the lab and then possibly provide a patch if the software had a problem.

This meant that we needed to keep working branches of prior releases available so that we could patch that particular release (without forcing the client to upgrade to the latest version).

"Big Bang" Release
As another example, at a large company, there was a project to replace an aging scheduling system. Scheduling tens of thousands of employees on a regular basis is difficult enough, but when you add the various rule sets that come from state regulation, company policy, and union contracts, things get complicated. (Fortunately, I was not on this particular project, but I did get to interact with a lot of people who were.)

This is where it becomes difficult to come up with the "minimum viable product" for the initial release. If you target one group of employees, you find out that their particular union covers more than just that group, so you need to expand the scope of employees included. If you manage to find an isolated group, you find that some of the employees move back and forth between departments (depending on how busy things are).

When you take the variables into account, the "minimum viable product" comes so close to "full replacement" that it doesn't make sense to do things incrementally.

And I'll be the first to say that this is a difficult way of doing things. There's no way to get everything right the first time (especially with this type of release), which means that the first six months after release are going to be a bit painful for the users and the development team.

Moving Into The Ridiculous
This can stray into the territory of "What the heck are you thinking?!?" I came across a group of business analysts who were collecting requirements for one of these large, company-wide projects. They were on a 24-month requirement gathering mission. Let me repeat that: 24-month requirement gathering mission.

That's 2 years! Even in a slow-moving corporation, things are going to change faster than that. By the time you get to the end of 2 years of requirements gathering, you'll just find that the business has changed, many of the features are no longer needed. And there will be a gap for features that are truly needed but came up too late to be included.

No Estimates - Continuous Delivery = ???
Now some of these examples are extreme, but it does bring up the question: can we get benefit from the principles behind No Estimates even if we are not in an environment where we can implement Continuous Delivery?

For example, in the quarterly release scenario, each of our clients had features that were important to them. It's not really possible to give each of them a "yes that's in this release" or "that will be in the next release" unless we are doing estimates on how long the various features will take, how much bandwidth we have from our development team, and what our planned release looks like. Definitely not a "No Estimates" mindset.

I don't have the answer to this. It may be that we need to come up with better ways of determining what the "minimum viable product" is. This is not too hard when working with green-field development. But when working on system-replacement projects, it's a bit harder to convince the business area that they need to stop using one function in the old system so that they can use it in the new system (especially when there is no practical way of actually disabling the feature in the old system).

It may be that we need to have better delivery mechanisms. If you are deploying just to your own servers, then it becomes much more practical to do weekly or daily releases. And there are some web-based companies that do multiple releases throughout each day.

This is a viable solution for server-based software that we run on our own servers or put into the cloud. It is more difficult if the software needs to be installed on servers at a client location. (And I would bet the majority of clients don't really like the idea of "auto-update" on their servers -- but that thinking will gradually change as people integrate more cloud solutions and see more and more successful updates.)

Wrap Up
As you can see, I've been thinking quite a bit about this. I really like the idea of No Estimates, development by subscription, and continuously adding value to the business. I think that I've identified a number of scenarios where this works well, and it really just requires buy-in from everyone involved.

But there are some scenarios where it doesn't seem to fit at all. I don't want to abandon the ideas (because they are good ones), and I'd like to see if there are small elements that we can bring in to these projects to improve the development process.

No tool is perfect for every job. But there's got to be some good principles that we can bring over into these other environments.

For more info, feel free to join the #NoEstimates discussion on Twitter, or look up Neil Killick or Woody Zuill. And I'll be glad to hear any of your insights on this topic.

Happy Coding!

More Organic Software: Escaping Into the Wild

A couple days ago, I wrote about how an application I wrote for myself escaped into the wild and became added value for the company. But I also had this happen with another piece of software that I wrote for myself. The way it grew organically based on business needs makes me think about how No Estimates can be appropriately applied in similar situations.

Working with Time Clocks
In addition to working with automated turnstiles, I also worked with the time clock system. Specifically, I worked with the time clock hardware and data collection. This was then transferred to the time tracking system (which was the "system of record" when it came to time tracking).

The time clocks worked with similar hardware as the turnstiles, and so I would check them every morning to make sure that the everything was online. Initially, I used a set of SQL scripts to check this, but as you can imagine, I ended up writing a little application to help me out with this.

This application had a drop-down box that listed all of the time clocks and a data grid to show the latest transactions. Again, I could easily scroll through the drop-down and see which time clocks were reporting data. This required a bit more thought, though -- some time clock locations were used more heavily than others. For example, the time clocks at the site entrances had a steady stream of transactions, but the time clocks that were in the maintenance shops were only used by a small group of people, so it was normal to have sporadic transactions.

Escaping into the Wild
The scheduling team would use the time tracking system throughout the day. They would make adjustments if someone clocked incorrectly (or forgot to clock in/out). And they would also check for "no shows" -- people who did not show up for a scheduled shift.

Because of this, the scheduling team was more dependent on having the data consistently sent to the time tracking system throughout the day.

One of the schedulers saw my application and said, "Can I have that?"

And the reason turned out to be pretty simple. If he saw a gap in the time tracking system, he could check to see if a particular time clock was off-line. If an employee regularly clocked in at the west entrance, he could check the 3 time clocks at that location to see if they were all reporting data.

If one of the time clocks appeared to be off-line, then the scheduler could contact my team to have it reset. This gave them better visibility to the system, and they knew that if a time clock was off-line, they would not necessarily mark "no shows" for people who normally clock at that location.

More Features
I think it's good when a business area asks for more features. It shows that they are thinking about the technology and data that we have provided to them and coming up with new ways to use it. So, I wasn't really surprised when I got a request for more features in the time clock viewer application.

We had security swipes at the entrances to the site. This would allow the security team to easily check that people were allowed to enter the property. The employees would swipe their card, and get a "go" or "no go" based on whether they were an active employee (cards would get automatically disabled if someone quit or was let go). But this also meant that people who needed to clock in had to swipe again at a time clock.

To streamline this process, we combined the security system and the time clocks. We used the time clock to collect the card swipe (either a bar code or RFID) and then pass the card information along to the security system for validation. This resulted in a single swipe for people who needed to clock in as well.

But there was a process issue: the employee would need to press the "Clock In" button the time clock before swiping his card. If you did not press the button, the security swipe would be processed and you would be allowed on property, but a time clock record would not be generated.

"I Forgot"
As you can imagine, when people do this everyday, sometimes they forget to push the button (as they are maybe rushing too quickly one day), or they think they pushed the button but it didn't register, or they simply forgot to check for the "clock in accepted" message that would normally display.

If someone forgot to clock in, the schedulers could manually add a record to the time tracking system. But because this was a common occurrence (especially when there are several thousand people clocking in throughout any particular day), the schedulers wanted a bit more information.

You see, some employees had figured out a way to game the system. If they got to the clocking location and found that they were 2 minutes late, they could "forget" to press the "Clock In" button and ask their scheduler to override it. This would mean they wouldn't get penalized for being late.

So the schedulers asked if there was a way for them to get the time clock data combined with the data from the security system. This way, they could look up an individual and see what time the security swipe went through. This would let them check to see if people were actually late when they came in, and it would allow them to separate the people who forgot to clock in from the people who "forgot" to clock in.

Business Changes - Things Get Complicated
Whenever you deal with time tracking, things get really complicated. There are federal and state regulations, there are company policies, and there are union rules -- all of which need to be followed.

One change that came about was the ability to clock in and out at your work location (instead of the property entrance). The reason is that it could take 10 to 15 minutes to get to your actual work location from the property entrance. You were supposed to be at your location and ready to work at your scheduled time, so having a time clock at the location would help better track that.

Unfortunately, the time tracking system didn't have this level of granularity. It kept track of "clock in" and "clock out", but not the particular time clock that was used.

So, we were asked to come up with some printable reports that could be used by the location managers based on the raw time clock data. These reports would allow a manager to see all of the clock transactions for employees in their area (based on their department identifier). The manager could double-check that their employees were clocking at the right locations, and they could coach employees who needed a reminder on procedures.

Later on, we extended the functionality of these reports to show "exceptions" -- such as people who clocked in but forgot to clock out, or for people who clocked in and out at different locations. This let the management teams focus on where specific problems lie.

And all of this helped the company comply with the various contractual rules and legal regulations that governed the time tracking of employees.

Wrap Up
So, here's another example of an application that grew organically as the business areas saw places where we could add value. And it's another good example of how No Estimates could work well in a changing environment.

And even though I've come up with some good examples of where No Estimates would be effective in projects I've worked on, I've also been thinking about some situations where it would not be a good choice. (I'll be writing about these soon.)

This doesn't bother me, though. No tool is perfect for every situation. I wouldn't expect that No Estimates would be a good choice for every situation, either. What's important is that we take a look at our current processes, determine what's working and what's not working, and look toward improving things where we can.

Happy Coding!

Organic Software: Spreading Usefulness

I've often mentioned that I've built the most successful software by understanding the users and their day-to-day business. But sometimes, I make something that is useful for myself, and it grows in unexpected ways.

Working with Turnstile Counts
I spent may years working for a major theme park. I actually worked at the company for 5 years before joining the I.T. team as a junior developer and hardware assistant. As an assistant, I worked with Doug (the technical lead) on automating the attraction turnstiles. Prior to automation, turnstile counts were collected on the device (usually a mechanical turnstile with an LCD screen), and the employees working at the location were responsible for writing down the counts from the LCD screen every hour. Then they would manually enter the counts into a tracking system.

The automated turnstiles were network-connected and sent their counts to a server every 15 minutes. This was great for the people working at the location because they no longer had to do the manual recording process. On top of that, we could automatically transfer the counts to the tracking system.

But there was a problem: the early versions of the network modules were a little flaky and would go off-line. None of the data would be lost, but the network module would need to be reset to send the data to the server.

One of my jobs was to check the turnstile controllers every morning. This involved running some SQL scripts against the database and looking for "gaps" in the data. If a particular controller had not reported data for a while, I knew that the controller needed to be reset.

Lazy Programmer - Productive Developer
As we converted more and more attractions, this process became a bit more tedious. When there are just a few controllers to look for, the gaps were very obvious. But when there are several dozen controllers, the gaps in the data are harder to find.

I'm a lazy programmer, and so I did what any lazy programmer would do when faced with a repetitive task: automate it.

Okay, so I didn't go full automation. But what I did was put together a simple application. It just had a drop-down box and a data grid.

The drop-down box had a list of all of the turnstile controllers. When a controller was selected, the grid would show the data for that one controller (with the most recent record on top). This made the job of checking the controllers very easy: I could just scroll through the drop-down box and look to see which items did not have current data.

The Application Escapes
One day I was reviewing the data with one of the managers from the area. When she saw my little helper application, she said, "Hey, what's that?"

I demoed the application and showed the raw data that came from the turnstile controllers. And she said, "Can I get that?"

The reason for this was simple: the manager had access to the tracking system which gave data that was one day old (yesterday's data). The little application that I had provided current data (well from the latest 15 minute data block, anyway). The manager really liked the idea of being able to access the data throughout the day. This way, it could be acted upon immediately.

For example, if the counts were zero, that may indicate that that the attraction was down (maybe due to a mechanical fault). If the counts were lower than expected, it could indicate that there is a staffing problem at the location that lowered the throughput. For either of these situations, the manager may want to go to the location to check what's going on.

So, I cleaned up the application a bit. I parsed the raw data into columns that the operations folks could easily understand. And I also added some total columns (for example, if there were 3 turnstiles at a location, they came through as separate records, but the operations folks were more interested in the total counts for the location. Another example, the counts came in 15 minute intervals, but they wanted to see hourly totals since that's what their operational metrics were based on).

Soon, there was an application that I had originally built for myself that was now being used by about 50 people to help them do their jobs better.

More Functionality
After the conversion from manual turnstiles to automated turnstiles was complete, this application now had all of the raw data for the locations. The managers and leads could check the data throughout the day.

As expected, the operations folks came up with some new ideas for the data. First was the idea of notifications.

At regular intervals throughout the day, they wanted to receive email or pages that had the current data. This meant that people working in the area (and not at a computer) could still take advantage of the data.

This wasn't difficult for us. We already had a notification system that could send out email or pages to different groups and individuals (we had lots of Blackberrys and Motorola pagers in use at the time). In fact, the biggest part of adding this task was to build some maintenance screens. Each person could choose which data they wanted (they could select just a few attractions or all of them) as well as the times they wanted to be notified (for example, every hour during normal operating hours) as well as the days of the week to be notified (so they would only get the information on days they were working).

This added about 50 more people who could access the data on a regular basis to help them keep the business functioning efficiently.

Yet Another Feature
The next step was to add "alerts". For each location, they could enter a target number and a warning threshold. For each page, we would include the current count plus the target number for reference. And if the current count fell below the warning threshold, that would trigger a separate set of "alert" pages.

And Another
The last major feature that we added was a summary email that was sent out every morning. This would include a summary of the prior day's data so that directors and area managers could track how things were going. This replaced a manual process that the attractions staff was doing.

Now the leadership team was able to directly benefit from the data collected.

Organically Grown
Now this isn't a process that happened overnight. These features grew over a period of several years. But the interesting thing about this is that it was not a set of features that was originally requested by the business area. It was something that I built for my own use that happened to have usefulness beyond just me.

And I think that's why I have such an affection for this particular application. This wasn't a "project" that the business area came up with. When the turnstiles were being automated, the primary benefit was to eliminate the labor spent collecting the counts manually (and the potential for transcription error) and putting the data into the tracking system (with potential for more errors).

No one asked for a system that would page (near) real-time numbers to the operating areas. But once we had the data, and the users in the business area started to see the how they could benefit from it, things just grew naturally.

We discovered value incrementally over time. And the end result had a lot more benefit than anyone had anticipated.

Wrap Up
So, this brings us back to the idea of "No Estimates". It's not just that we are bad at estimating features that are asked for by the business area. It's that we don't even know where the real value lies when we start out on a project.

This really shouldn't be a surprise. One of the major tenets of Agile is that we adjust our direction when we need to. When we see potential value, we're allowed to shift our current direction to pursue that. And these feature requests came in small chunks. At the beginning, I never would have been able to put together a viable estimate/plan for "send out current turnstile counts to selected individuals". But because we took things one step at a time (raw data, formatted data, summarized data, notifications), we were able to get to this unexpected result that added value to the company.

I guess I'm convincing myself more and more that I like the idea of No Estimates. When I look at projects that added real value to the company (like this one), it seems like we pretty much worked that way. (There was some project overhead, but minimal compared to most of the applications we worked on.)

I've got a couple more stories like this one that I'll be sharing soon. In the meantime, keep thinking about ways that you can be an effective developer and add the most value to the company.

Happy Coding!

No Estimates: Development by Subscription

Last week, I spent a couple of days at Agile Open SoCal. It was a great time to talk about various issues and successes and get some great advice from other people who are in similar situations. There's a lot for me to process, and a couple of topics really stuck out.

One of these was No Estimates. I'll have to admit that I've been following this discussion from the sidelines. If you want to catch up on what No Estimates is, check out what Neil Killick and Woody Zuill have to say. I will not do the description justice, so you'll want to go to these guys who have been thinking about it for a long time.

I know Woody from the SoCal developer community, and I had a chance to sit down and talk to him a bit last week. (He's also known for showing the success his team has had with Mob Programming).

We Suck at Estimation
One of the basic truths is that we (as developers) suck at estimation. But that's not really a surprise. We are constantly being asked to do things that we've never done before.

An experienced cook will be able to tell you how long it will take a bake a cake. After all, he's done it a dozen times and knows what to expect. But for someone who's never baked a cake (like me), I would end up "padding the estimate" on how long it will take me. Is it because I don't know how to follow a recipe? No. It's because I'm working with unfamiliar tools -- I don't know exactly how to use the mixer or if I'm going to mess up a batch by over-mixing or under-mixing.

And this is true in software. Whenever I'm looking at a new project, it usually contains 4 things I've done before and 1 thing that's new to me. Now, I'm not completely ignorant of the technology (I've seen someone use the mixer), but I don't know what problems to expect (if you don't grease the bowl before you start, you won't get an even mixture).

I do not like to not deliver on my commitments -- meaning, I would rather over-estimate and deliver early than under-estimate and deliver late. The same is true in the rest of my life: I'd rather arrive at an event an hour early than 5 minutes late.

This means that I have a tendency to over-estimate:



I don't do this so that people will think of me as a miracle worker. I do this because I want to make sure I deliver on my commitments.

But this doesn't make my estimates any better than someone who under-estimates. In both cases, our estimates are wrong.

Eliminating Estimates
So, the idea of No Estimates is that we get rid of the estimation process. Instead of coming up with a huge project plan with lots of incorrect estimates (that end up compounding the problem), we focus on continually delivering small pieces of value.

I like the idea of this. And as a responsible developer, I think that it would work well for me and my development style -- I like to make my users happy, and I can best do that by constantly giving them features that make their jobs easier.

But I have been skeptical for a number of reasons. First, not all developers can work this way. There are some developers who would take advantage of the situation and slow down their delivery. There's no longer things like "story points" and "velocity" that measure productivity. And even though those measurements are probably wrong, people like to have something tangible to look at. The solution to this is to have managers who can keep track of productivity (in other ways) and know the capabilities and output of their teams.

Second, many companies cannot work this way due to their budgeting process. The financial systems are built around large projects and schedules. Marketing wants this and Sales wants that and Operations needs this other thing. So let's figure out how long each thing will take, how much our development team can handle at one time, and then prioritize everything. On top of that, money for the projects comes out of different buckets depending on whether it is a capital or expense project. I have no idea how to resolve this issue.

No Estimates is more of a subscription model for development. Instead of the business areas putting aside money for Project X that will cost $50,000 (maybe), the development team is funded on an on-going basis -- just like the other teams (finance, sales, operations, maintenance) that add value to the company.

The Business Areas Already Do This
I had a realization about this based on my experience: even though the company does not work this way, the business areas do.

I worked at a large company (20,000 people on site, 100,000 people in the division), and I built software for the various business areas based on what came down from project planning. And this was a hassle for many of the business areas. Sometimes a department just needed a small piece of functionality. It wasn't large enough to justify an entire project -- maybe just a report to collate data. And even if it were big enough, it probably would not be prioritized near the top of the list. So it may take a year or more to get it on the schedule.

The solution: Shadow IT. I don't know if there's an industry name for this phenomenon, but this is what we called it in our company. A department would have someone on staff who was more technical than most and probably good at something like Microsoft Access. This person would then build all of these small pieces of functionality that the department wanted to make their job easier.

So the business areas have already bought into the idea of development by subscription. They have decided that it was worthwhile to pay someone in their department to handle these types of things. And this Shadow IT person was paid out of the same budget as all of the other employees in the department.

(And this arrangement isn't all puppies and rainbows. Since the majority of the Shadow IT people did not have any formal technical training or experience, some of the projects would run into scaling issues or simply stop working when the data got too large. At this point, the systems were critical to the department, so they would often turn to the development team to fix the issues or rewrite the system.)

So the business areas have recognized this as a way to add value to the company. Now we just need to get the company itself to realize it.

This would give the opportunity for the business areas to have access to actual developers who can deliver functionality quickly in a supportable way.

Wrap Up
I really like the idea of No Estimates, and I hope that it gains traction. There is a lot of trust required between the company and the development teams (but I've wanted to have better integration between the company and the development teams for a long time -- that trust is needed anyway).

It was an eye-opener when I realized that the business areas had already implemented their own version of No Estimates by having a Shadow IT person on staff. Next, we need to figure out how to formalize this so that the right people (developers) are doing the right work.

Happy Coding!

Advice for New Developers (and Experienced Developers)

I've had several developers contact me recently asking for advice on getting into a development career and how best to learn about programming. I'm not sure that I'm the best person to ask since I sort of lucked into my career and got great mentoring and training as a junior developer, but here are my thoughts.

Not Just New Developers
Before we dive into this, I want to point out that this advice does not apply only to new developers. Our industry is constantly changing, which means that we need to be constantly learning. And experienced developers need to figure out new technologies and where we want to concentrate our efforts. So we get to keep going through this process every few years (or less).

Find What You Love
The first thing to do is figure out what technologies you need to learn. Something that we need to realize is that we can't learn everything. If you try to do everything, then you'll eventually find that you're competent in nothing. It's best to concentrate on one or two key areas where you can build a strong understanding, and then get a peripheral understanding of other technologies that are available in that environment.

Pick an environment or technology that you love. This is something that took me a long time to learn. Life is short, and work takes up a big part of that time. Do not do something that will make you miserable. There was a time in my career where I dreaded getting out of bed in the morning just thinking about all the things that I had to do. It's totally not worth it.

Finding what you love may be based on a particular industry. Maybe you want to be in entertainment or finance or medical devices. If this is the case, then investigate the companies that are working in that industry. Find out what technologies they use. Most industries gravitate toward similar technologies. For example, functional programming is big in the finance industry; embedded programming is big in the medical device industry. If you are working in an industry that you are passionate about, then you'll want to be productive and motivated to learn the necessary technologies.

Finding what you love may be based on a particular technology. Maybe you find that you love to tinker with Raspberry Pi devices; maybe you like to understand how humans work and want to make user interaction experiences better. At the same time, check to see how the technologies are used in the real world. We have to be practical. It may be that our favorite technology does not have many career prospects. But we can usually find something similar that is widely used in some industry.

I'm going through this now. I really love XAML and WPF, and even though it's a viable (and awesome) technology, it seems to be starting to decline (at least in the desktop world). There are times when I feel like I should be diving into JavaScript and the plethora of libraries, but that does not make me very happy for a variety of reasons.

Conversely, I've been exploring functional programming through different languages, and that has me much more interested. I've found that I've been using functional concepts without realizing it, so this seems like a more natural fit for me.

Find How You Learn Best
As humans, we learn in different ways. Some people learn best from books, some from videos, some from working with a mentor, some through formal training, some through reading the documentation, some through labs, and some through tinkering around.

Try a couple different learning techniques, and find which one works for you. Not everyone learns the same way. So, try a couple different techniques. I've found that I learn best through books. It lets me easily go back to previous topics, re-read sections, pause and concentrate on a particular topic. When I watch video, my mind tends to wander or I find that I want to open up an IDE and try some stuff out. So I usually find myself re-playing parts of a video over and over. (I say "re-playing" rather than "re-watching" because I really wasn't watching it the first time.)

But that's just me. Explore and find what works for you.

Get Involved
The best thing we can do is talk to other developers, especially when we're exploring new technologies. Find someone who's been working with the technology in the real world. You can get some great advice from that person: what works, what doesn't work, the things that are awesome, the "gotchas" that you have to work around. Talk to other people.

Get involved with a local user group (or several user groups). This has been a huge eye-opener for me. I was pretty late to the game; I've only been involved in the developer community for about 4-1/2 years. Before that, my developer interaction was limited to the people that I worked with. I was fortunate to work with some really good people, but I was missing out on a much, much broader community.

What I've found is that developers love to talk to other people, but (stereotypically) we don't want to *start* the conversation. If you're new to a group or are with a bunch of developers you don't know, just go up and introduce yourself to someone. Ask about what that developer does, and you'll find that a conversation gets started. And usually, other people will come over to join the conversation. I've seen this happen many times; I just had to be brave enough to get the conversation started.

If you don't have a user group in your area, start one. It might just be putting something on MeetUp to have coffee with a couple other developers. And it doesn't matter if you're working with the same technologies are not. Just start talking to other people.

In addition to getting advice on technologies, you'll also build relationships. Over time, you get to know a variety of developers and they get to know you. When you're looking for work, this is invaluable. After being involved in the developer community for a few years, I found myself looking for a new job. I was amazed at how many job prospects I got through people that I knew in the community. Not all of these prospects will work out, but you only need one job (hopefully).

Wrap Up
As developers, our industry is constantly changing. That means we need to be constantly looking at new technologies and we need to be constantly learning.

It is a challenge, but it is also exciting. We get to do new things all the time. We get to discover new ways to meet the needs of our business and users. We get to create on a daily basis. We get to keep learning. That's a pretty awesome job to have.

Happy Coding!