I ran across this today and thought I’d comment briefly about it. How can you interpret inaccurate estimates? What might it mean when you estimate incorrectly?

I know individuals that beat themselves up over inaccurate estimates. I know teams that beat each other up. I know managers that beat up their teams. I even know a company that awards bonus compensation based on the accuracy of their programmers’ estimates. In those environments, inaccurate estimates hurt, so although I don’t like cost estimates in general for most teams most of the time, I recognize that some people must play the game to keep their job. That said, I return to something Ward Cunningham said in the early days about Fit: when we look at test results in a spreadsheet, we can look at patterns of red cells to learn something about our patterns of failure. He drew our attention to two kinds of patterns: systematic failure and sporadic failure.

In the early XP literature, books like Planning Extreme Programming told us that by measuring velocity we gracefully handle systematic estimate inaccuracy as the “value” of our story points fluctuates over time. Think of it like a floating currency: its value in hours changes slowly over time as our ability to complete points improves. If you prefer to estimate in hours, then after several months, you might notice a bunch of stories whose actual costs were close to some constant multiplier of the estimate. In that case, you would consider multiplying all remaining estimates by that constant until you internalized it and began estimating in the “new scale” out of habit. Even when I cared deeply about cost estimates, I worried little about systematic inaccuracies.

Sporadic inaccuracy hurts more. Since we provide cost estimates to that others might plan, we owe it to them to reduce the uncertainty of the actual cost of our work, in order to make planning more useful. I have developed a conjecture over the last few years that changes the question of sporadic estimate inaccuracy:

The actual cost of a story depends on (among other things) the level of complication of the story and the current state of the design. The higher our technical debt, the more that cost dominates the cost of the story.

I have noticed various people floating the definition of “technical debt” to suit their needs, so I want to clarify what I mean by “technical debt”:

By technical debt I refer to the latent cost of the amount of rot in the design. I think of technical debt as the cost of the design improvements we need to make in order to feel comfortable adding features or fixing defects in that part of the system.

You can think of technical debt as financial debt: interest-bearing principal owed to another party. In this case, the other party is the system or the project, the principal is the current design and the interest is the extra cost associated with either rescuing or working around the current design. With these definitions established, I can state my conjecture this way:

In systems with high technical debt, the cost of repaying that technical debt dominates the cost of a story.

Since not all stories affects all parts of the design uniformly, we can do a little better:

The cost of a story depends on the level of complication of the story and the amount of technical debt in the areas of the design we need to change or extend to deliver the story. Working in areas with high technical debt causes the cost of repaying that debt to dominate the cost of the story.

I think you get the point. From this it follows that in systems with generally high technical debt, the distribution of technical debt effectively determines the uncertainty in the cost of the stories. Sporadic estimate inaccuracy, then, likely has a clear root cause: high technical debt distributed decidedly non-uniformly. This follows logically, because if the system distributed technical debt uniformly, then our estimates would show systematic inaccuracy.

This allows me to make two broad claims:

  1. In general, if a system has high and sporadic technical debt, we’ll tend to estimate with sporadic inaccuracy even if we estimate the relative difficulty of those stories with perfect accuracy.
  2. In general, we can expect to estimate with at worst systematic inaccuracy when delivering stories for a greenfield system or component.

I can interpret these claims more pithily:

  1. When working in legacy code, blame wacky estimates on the code base.
  2. When working in mostly clean code, blame wacky estimates on the programmers.

In particular, if you feel bad because you can't seem to estimate accurately when adding features to a legacy system, you can relax. As you attempt to build those features with high discipline, the resulting uncertainty in your actual costs will come from the current state of the design. The design will actually try to convince you to cut corners. Cutting corners can only improve the accuracy of your estimate for the current story at the expense of the remaining stories. Cutting corners can only get your project manager off your back for a day or two. You will eventually need to stop cutting corners.

References

Kevin Schlabach, "Help! Our Estimate Was Off By a Lot!".

J. B. Rainsberger, "The Eternal Struggle Between Business and Programmers" The article that has become the successor to this one.