Project failure matters. It costs billions. It reduces business competitiveness, agility, and sometimes even leads to business failure. That leads to job loss. It affects the economy. That’s why everyone should care about it.

Whether you’re a CEO or interning as a software developer, software project failure matters a lot¹. In this first post in the ‘We’ve got it all wrong’ series, I want to explore the key reasons I think we see so much project failure in the industry.

Failure flavours

Before I go any further, just a bit about what I’m not covering. Computer Weekly regularly features epic horror story projects that die slow deaths after massive spend. These cross-organisation, multi-vendor, massive, hope-it-all-meets-in-the-middle type of projects are different beasts entirely. I’m not covering these today. Maybe some other time.

Today I want to stick with the everyday, bread and butter, projects that are the mainstay of the industry. For the sake of discussion, I’m going to group project failures into two categories:

1. Canceled projects
2. Projects that fail to deliver significantly on what was promised

The canceled projects category is self-describing, I hope. Type 2 failures come in a variety of flavours including:

• very late delivery
• legacy software on creation - brittle, non-extensible, gaffer-tape systems that can’t support business agility
• subset of specified functionality

I’ve seen a few type 1 failures and a lot of type 2 failures. I’m going to make a statement that some of you may not agree with. I’ll make it anyway, and you can let me know what you think. I want to argue that these failures result from two core reasons:

• People Failure - the failure of an individual to successfully operate in a given role
• Process Failure - the failure to define or execute an effective software development process

From my experience, of the two, People Failure is the most common and detrimental. In fact with the right people, operating well, in the key roles you tend to end up with solid process.

People Failure

Not all roles are created equal. The failure of key individuals can have a much more significant impact than non-key roles. E.g. if a junior developer fails you may get some bugs, have to rewrite some code, lose a week or two. If your senior technical leader fails the project may prove non-viable due to bad architecture and failure to address key technical issues. If your key managers fail the project may not be delivered at all.

Most organisations do not address this factor. In fact, as an industry, I would argue that we exacerbate the issue. I think there are two primary industry practices that fuel this issue:

1. We don’t develop technical leaders.
2. We hire and forget.

Arrested Development

There is a huge shortage of technical leaders in the industry. It’s no wonder, given that most organisations make no significant effort to develop them. We kind of seem to expect them to grow on trees.

In fact, it’s a wider problem. I’ve been involved in technical interviewing for over a decade. One thing which has not changed in this time is that it is still really hard to find good software developers. That is because we don’t develop them either!

Hire and forget

As an industry, we interview people, deploy them and then just assume they are going to do okay. Sometimes they fail. They kill our projects. Utterly. It’s not their fault. It’s ours.

Generally people failure does not even make it onto the project’s risk register. I’d argue that on most projects it is the number one risk. Simple as. Yet most organisations do not manage it. Wow.

In fact, most organisations don’t even understand the difference between a good tech leader on a 5 developer project and one on a 50 developer project. And even if they do, often there just are not enough technical leads around for them to be picky.

I think the development of great developers and technical leaders is one of the key challenges that we face as an industry. This is something I’m sure I’ll talk about more in future posts, as it’s something I feel very strongly about.

Process Failure

Process is supposed to help build software. The way we do process, it often doesn’t help much at all.

Everyone loves to hate waterfall , but people did deliver software using it. Really, they did. I’m old enough to remember. In fact, I’ve never seen stats that prove we get better software deliveries since we started using agile and iterative approaches.

Don’t get me wrong, I’m a big agile advocate. I’m just not sure it’s scratching where we’re itching. I don’t think the issue is so much the flavour of process most organisations are using, so much as their fundamental relationship with process.

I think our difficulty with process is best summed up by the RUP/SCRUM/Kanban mix that has been seen in many organisations in recent years. There’s a lot of confusion out there.

It feels like people think agile is some kind of magic pixie dust. Let’s just sprinkle a bit of that on our project and we’ll be okay. Well, you won’t. Because it ain’t magic pixie dust.

I think we need to go back to basics. I’ll probably talk more about this next time.

It's time we grew up

People have been tackling traditional engineering projects for thousands of years. We’ve been doing software for decades. Some immaturity in the industry is acceptable.

However, there are problems we don’t need to live with anymore. I think we could build better software, quicker and have more fun doing it.

It’s time to get our house in order. It’s time, as an industry, that we grew up.