People keep forgetting that software development is always uncertain, because the minute you have certainty you’re no longer doing software development!

I like your idea of getting better estimates after some iterations. That must help a lot.

John (jfbauer), I love the slightly Machiavellian way you calculate your estimates. An incredibly non-cynical interpretation, and a useful one, is that you really do need to plan on those hard dependencies, and there’s nothing wrong with that at all. For example, right at this moment I’m held up a little bit on a software project because there are some issues I can only resolve by testing on certain hardware… that does not exist yet. I didn’t plan on that! And now I’m technically behind, although the client is being cool about it.

But the tactic you describe kind of reminds me of the way escape artists [are said to] flex their muscles and wrists to create a lot of wiggle room after they’re apparently tied up as tightly as possible. Sure, it looks like you’re on the most aggressive schedule that can be imagined, but you’ve bought yourself that wiggle room. Good thinking! Slightly deceptive, but good.

Bull, yes, you’re exactly right. One of the things I like about Scrum is that it’s really explicit about that: You get the best possible software, in a usable state, under the budget they give you. If the client thinks further development is worth the cost, you get more money or other resources. If not, hey, it’s all good.

That avoids the problem that occurs when software is ostensibly 90% done, but useless, and there is no money to finish it. I just realized this is a case of software’s infinite malleability being an advantage in estimation, which it usually isn’t.

Thank you all for your comments. I’d love to discuss this further.

This isn’t a new problem and it will never go away either. I think it is the mettle of the mature and successful CIO/CTO that they constantly “upward and sideways” manage their peers and management in general to get them to understand realities here. I’ve spent a good portion of my time on the job in these roles doing this kind of education and ongoing negotiation on “rules of engagement” etc.

I’ve written about how estimating is, however, a necessary evil. (In other words, the approach that WON’T work is to avoid giving estimates altogether). You can find my post, “Rock and a hard place: Why estimating turns into a squeeze play”, at http://www.peterkretzman.com/2007/11/15/rock-and-a-hard-place-why-estimating-turns-into-a-squeeze-play/

A reference in that post is to Construx’ software’s “Cone of Uncertainty”, which I literally keep framed on my wall, and point to it during meetings with key stakeholders as I evangelize these concepts. You can find that at http://www.construx.com/Page.aspx?hid=1648

Stakeholders are required to take on some of the risk in complex projects. To deliver quality software, the estimate may be exceeded and that is the risk the stakeholder assumes and it should be outlined in the estimate along with many assumptions about the project. The better your estimate the more confidence there will be in your ability to estimate. The better you deliver quality software on time, the less your estimate will matter.

The sales staff must work with the development staff to determine if we must meet the budget constraints or if it is more important to have quality software. Some stakeholders want the Cadillac, and some want the Pinto. Many times this can be determined by the nature of the project. There is no perfect solution and that’s what makes us valuable.

Assumptions and risks to go along with your estimate are key to controlling cost. Along with a sane stakeholder.

This post was mentioned on Twitter by MarkWSchumann: “Programmers are optimists” is a cheap shot, and here’s why. New blog post: http://bit.ly/A7eaa #agile…