Programmer optimism, and the “Death March”

I’m tired of hearing lines like “programmers are such optimists,” said in a disparaging way, by people who should know better. It’s a common refrain in corporate offices, usually when a software project is running late. Often the blame is placed on the programmers who are working for free on nights and weekends to make stuff work.

The project isn’t late because developers are braggarts. It’s because management has assigned them too much to do, which, as Peter Kretzman warns is the best way to not get what you want from your I.T. organization.

The “optimist” accusation is a cheap shot, and here’s why.

Here’s the project kickoff meeting.

It’s for some strategic initiative that executive management is excited about. Once in a while, the initiative itself is kind of stupid, but for argument’s sake let’s say it’s actually a reasonable idea: we’re opening retail stores in an underserved market, adapting to a new regulatory regime, or introducing a new online product. Stuff like that.

This meeting is led by, let’s say, the CFO, who has already budgeted the project to the departmental level. Line executives under the CFO are making fifteen-minute presentations (with Power Point! Yay!) on their areas of responsibility. Every slide has bullet points for tasks, deliverables, and objectives. And then department heads break out with their teams to figure out how to make it all work. It’s a half-day or full-day exercise.

Team building! That’s great.

Now let’s say you’re in the middleware group. You’re all about the transactions and the data interfaces, right? And obviously at this point nobody’s defined a transaction set. You know it’s going to be Oracle, and the database developers are in the next group over, and they have to work hand-in-glove with the DBAs before you get any information from them anyway.

You have just seen a half dozen slides full of bullet-point tasks.

And they want a time estimate for each one. In hour terms.

That’s not crazy.

But it is crazymaking.

Of course management wants an estimate of hours. They need to take that estimate, load-factor it, and divide by something like 160 to figure the number of person-months your area’s going to need. Very logical, and quite honestly that’s the best they can do at this point.

But raise your hand if this has ever happened to you… or never mind, raise your hand if it hasn’t: You give them an hourly number that calculates out to 36 person-months, and there are four of you on the team, and so you figure that’s nine months, and they roll out the top-level Gantt charts showing your part of the project doesn’t get the ball until June, but we go live before Thanksgiving.

Naturally it’s all critical path. You don’t see slack before or after your slot on the Gantt.

Okay, that is crazy.

You’ve been in this racket long enough to know this is pushback time. You go to your team lead or your department head, on the spot, and say it. “We can’t do this. It’s nine months with everyone fully committed, and that’s not taking into account our maintenance load on everything else. They’re giving us five months.”

Then the haggling begins.

So far everything I’ve said is par for the course in classic project management, but here’s the point that people don’t seem to talk about. While it’s true that management is unqualified to dictate five calendar months or 5000 person-hours, so are you.

This is where the folk legend of programmers as optimists comes from. You gave your estimate at the kickoff meeting, but it was based on all you had at the time: bullet points and a good knowledge of your team’s abilities. Unfortunately, the project doesn’t really run on bullet points, and when management responds to your pushback, what are you going to say?

By contrast, Purchasing can say flatly that the lead time from China is six months no matter what. You can’t make the assembly contractors and ships go any faster. Finance can say with confidence that cash flow absolutely will not support a bank loan until September unless the board is willing to pay twice the interest. Logistics has a warehousing model. Marketing has stats to show that the product rollout campaign is most effective in August and is simply impossible before July anyway.

And what do you have again?

You have bullet points and what’s in your head. (I’m deliberately putting the words “head” and “bullet” close together here.)

To say that “programmers are optimists” is foolish at best. The reality is that programmers work with the most malleable part of the process. It’s true! Virtually everything we do is mental work. Even when we need “tools” they are usually the kind of thing you can order online and download in fifteen minutes.

Other specialities have the luxury of outside factors that buy time. We don’t.

So there you go.

The CFO calls her department heads in for a status meeting two weeks later. Costs for this initiative don’t fit into the budget the board approved, so 20% has to be cut. She can’t cut production costs any further, it’s already being done in China. She can’t cut financing costs. She can’t cut the construction costs because it’s already low bidder. The only place to cut time and money is where the estimates were vague and uncertain to begin with–which is you.

Stuff rolls downhill, as it tends to do, and your manager starts the Monday meeting with the announcement that the new project is simply going to have to get done in five months, not nine, with no additional resources. Can you say no to that? On what basis? Can you really argue that the bullet point “Create JSON interfaces to SQL data” is definitely going to take the 200 hours you said rather than the 80 they want you to say?

Of course not. It’s a bullet point. You’re on your way to what Yourdon famously called a “Death March” project.

Guess what.

You’re basically getting a project with “Agile” inputs and “waterfall” outputs. In this scenario, you don’t have adequate information even to make a time estimate, until it’s time to dig in. This isn’t actually bad in its own right–it’s kind of Agile in nature–but you were committed to some other, lower estimate six months ago and it can’t be increased without breaking critical path.

At this writing, I don’t have a great solution to the problem of timeline crunch. What I do know is that locking in a time schedule with nonnegotiable objectives is a near-guarantee of failure. If you make it succeed, it’s against the odds and probably your health, and if it fails… that’s not your fault.

How do you prevent Death March projects in your I.T. department? The comment section is yours.

About these ads

12 responses

  1. Social comments and analytics for this post…

    This post was mentioned on Twitter by MarkWSchumann: “Programmers are optimists” is a cheap shot, and here’s why. New blog post: http://bit.ly/A7eaa #agile…

  2. Spot on with Agile inputs and waterfall outputs. The only trick I’ve used is to any degree of success is creating plausible “pre-tasks” that slow down the estimation time to completion to allow for some possible negotiation prior to estimates being cast in stone. It doesn’t work in the exact scenario described where the implementation date is already determined essentially before a work estimation is actually requested. Published SLAs for commodity work do little to buy more excuse and “I told you so” arguments, but without SLAs you have nothing. I’ve written a little on this “pre-task” concept here: http://bit.ly/Sw8y8

  3. The stakeholder always wants to know how much the project is going to cost before you deliver it. This is true with all complex manufacturing type projects whether it is skyscrapers, bridges, or software. All an architect can do is give an estimate and make sure the stakeholder understands that if they want quality software that the estimate is just an educated guess that provides a budget range. It is NOT a contract.

    Stakeholders are required to take on some of the risk in complex projects. To deliver quality software, the estimate may be exceeded and that is the risk the stakeholder assumes and it should be outlined in the estimate along with many assumptions about the project. The better your estimate the more confidence there will be in your ability to estimate. The better you deliver quality software on time, the less your estimate will matter.

    The sales staff must work with the development staff to determine if we must meet the budget constraints or if it is more important to have quality software. Some stakeholders want the Cadillac, and some want the Pinto. Many times this can be determined by the nature of the project. There is no perfect solution and that’s what makes us valuable.

    Assumptions and risks to go along with your estimate are key to controlling cost. Along with a sane stakeholder.

  4. Great post, Mark, and excellent comments from people so far as well.

    This isn’t a new problem and it will never go away either. I think it is the mettle of the mature and successful CIO/CTO that they constantly “upward and sideways” manage their peers and management in general to get them to understand realities here. I’ve spent a good portion of my time on the job in these roles doing this kind of education and ongoing negotiation on “rules of engagement” etc.

    I’ve written about how estimating is, however, a necessary evil. (In other words, the approach that WON’T work is to avoid giving estimates altogether). You can find my post, “Rock and a hard place: Why estimating turns into a squeeze play”, at http://www.peterkretzman.com/2007/11/15/rock-and-a-hard-place-why-estimating-turns-into-a-squeeze-play/

    A reference in that post is to Construx’ software’s “Cone of Uncertainty”, which I literally keep framed on my wall, and point to it during meetings with key stakeholders as I evangelize these concepts. You can find that at http://www.construx.com/Page.aspx?hid=1648

  5. [...] you work in a shop where obvious refactorings are “a waste of time”? Do they turn your best estimates to mush? Is it hard to do your best work because they won’t let you fnish things? I can’t [...]

  6. Peter, I think I saw the Cone of Uncertainty, or something a lot like it, in some of Steve McConnell’s work… maybe Rapid Development, not sure. Same concept: you get better and better at predicting project outcomes the closer you are to the end.

    John (jfbauer), I love the slightly Machiavellian way you calculate your estimates. An incredibly non-cynical interpretation, and a useful one, is that you really do need to plan on those hard dependencies, and there’s nothing wrong with that at all. For example, right at this moment I’m held up a little bit on a software project because there are some issues I can only resolve by testing on certain hardware… that does not exist yet. I didn’t plan on that! And now I’m technically behind, although the client is being cool about it.

    But the tactic you describe kind of reminds me of the way escape artists [are said to] flex their muscles and wrists to create a lot of wiggle room after they’re apparently tied up as tightly as possible. Sure, it looks like you’re on the most aggressive schedule that can be imagined, but you’ve bought yourself that wiggle room. Good thinking! Slightly deceptive, but good. :-)

    Bull, yes, you’re exactly right. One of the things I like about Scrum is that it’s really explicit about that: You get the best possible software, in a usable state, under the budget they give you. If the client thinks further development is worth the cost, you get more money or other resources. If not, hey, it’s all good.

    That avoids the problem that occurs when software is ostensibly 90% done, but useless, and there is no money to finish it. I just realized this is a case of software’s infinite malleability being an advantage in estimation, which it usually isn’t.

    Thank you all for your comments. I’d love to discuss this further.

  7. If you really have to estimate the complete project within a day you’re really screwed. What we try is to keep our teams together and try to assign work to teams (not teams to work). That way we have at least a known velocity and are up and running with a reasonable estimate after a few short iterations (2-4 weeks). But this is definitely not possible in all environments…

  8. Thank you for your comment, Matthias. I agree that estimating in one day is problematic, although it’s not too crazy if you go with the Cone of Uncertainty model. That way you can keep adjusting as you learn more.

    People keep forgetting that software development is always uncertain, because the minute you have certainty you’re no longer doing software development!

    I like your idea of getting better estimates after some iterations. That must help a lot.

  9. While I don’t manage software projects, I often manage projects that involve software developers. (Technical training/learning for software products.) You’ve identified a key issue here, in the idea that those of us who aren’t unloading crates with forklifts are the ones to take it in the shorts when someone wants to tighten up their little Gant Chart.

    One of my tricks is to define my little part of the world in terms of components, rather than one big rock. On the software side, you might think of this as features or compatibility or speed.

    Then, when a pointy-haired boss tells me he wants it done in half the time, I just draw a big red “X” through half of the components and ship it back to him for a confirming signature. (No, I’m not seriously saying I want you to build software that’s only compatible with half the hardware — it’s a negotiating tool.)

    This often opens a meaningful dialogue about how many hamsters we have, how fast their little treadmills go, and the fact that decisions must be made. Or, at the worst, I’m able to limit the scope of work and be seen as having met my commitments — even if the product won’t run on the C-64 as they originally wanted.

    Not a perfect plan, but it kept my reviews clean and (now that I’m a contractor) it really keeps my client relationships MUCH more professional.

  10. @Dick (“not a perfect plan…”): Right, because you’re still up against the idea that software time is malleable and compressible. By breaking your part into chunks you give yourself wiggle room to negotiate things away, but the boss or client can still bicker with those smaller individual task estimates the same way they wanted to manipulate the overall estimate to begin with.

    But you’re right, it can be an effective technique.

    You know, right now I’m thinking of a rhetorical possibility. What if you (or I) said to the project owner something like this:

    I’m telling you it takes three months because it takes three months. If I back down now and say it’s six weeks, what will you do when we’re eight weeks in, it’s not done, and the stakeholders want an explanation?

    Basically, I’m making the transparency argument. If you lead with balderdash, how can you effectively explain the (inevitable) problems later?

  11. [...] Man, does that clear my head. It also avoids big time crunches when “something comes up” and I didn’t leave time for the urgently required stuff. Mark W Schumann´s last blog ..Programmer optimism, and the “Death March” [...]

  12. This week I turned down a project that I’d worked for three days on, and that I was being paid £1,650 a week to deliver. I turned down the rest of the project (and the money that would have gone with it) for exactly the reasons stated in this article. The project had poorly-defined requirements that only existed in someone else’s head (and probably not even there). And timescales were being set without properly estimating the actual time required to implement those (poorly understood) requirements.

    I’ve been doing what I do a long time now. I have somewhere in the region of fifty successfully-implemented projects behind me. That history of successful delivery is my lifeline. The fact that past clients have good things to say about me because of the results I’ve been able to achieve for them when they gave me what it was I needed from them is what gets me my next project. And the one after that. Success is a self-perpetuating habit, and so is failure. Whenever I encounter a death march project, therefore, I’m only too happy to walk away rather than be associated with someone else’s failure.

    In my experience, it’s usually not programmers that are overly-optimistic. Generally it’s parties that don’t actually code themselves, but that nonetheless have a vested interest in promising unrealistic delivery targets to their stakeholders. Whenever I encounter such people on a live project (which is rare these days, since I have a nose for them and usually manage to filter out these opportunities at the discussion/interview stage), my approach is to explain reality to them. Once. If they change their approach and take on board my advice, fine. Mostly I find they do not take on board my one warning shot. I generally find that people who prefer for political or cultural reasons to believe in fantasy outcomes that are not actually achievable know full well that they are being unrealistic. When I try to have “that conversation” with them they attempt to use it as an opportunity to sweep any concerns under the carpet and carry right on believing whatever fantasy they have a vested interest in promoting. When that happens, I find it best to let someone else disappoint them. If they’re hell-bent on failure, they can achieve that all on their own. They don’t need me for that.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 1,457 other followers

%d bloggers like this: