You Aren't Doing Agile

Stop me if you’ve heard this one before.

You’re starting a new job in software and you ask someone from your new team about their process and how work is prioritized. They say something about being an Agile team and working in sprints (one or two weeks in length, usually), but you look at the backlog and see tickets that won’t be worked on for several months with specific acceptance criteria and story points attached, linked to some sort of planning document that stretches out for months or even years.

Congratulations - you aren’t doing Agile!

Agile software development is frequently misunderstood by those who claim to practice it. Like crypto and AI, agile became a buzzword that companies could use to get interest from investors, job candidates, and tech writers. It quickly became synonymous with specific practices* and technologies, similar to crypto with Bitcoin and AI with LLMs/ChatGPT. It has an ecosystem of certifications that may or may not have any real connection to the underlying subject. The main difference is that you can practice agile without consuming massive amounts of electricity and straining local power grids.

When I was first introduced to agile/Scrum, it had been hyped up by one of my coworkers as a life-changing difference in how software was made. So I was disappointed to find in that introduction that we covered what could (boringly) be described as a new project management style that mostly consisted of adding more meetings. Having done waterfall projects before, I honestly didn’t really see the difference when it was presented this way. Yes, in agile you block out a couple weeks of work for a sprint and then check in at the end to see how it went. So? We had check-in meetings in our waterfall projects every week or two as well. We had projects broken down into chunks of work that could be split to different developers. Was the only difference in agile that we’d call those tasks “stories” instead?

Well, no. It took me a long time to understand what agile is actually about, and I had to pick up a few scars along the way before it really sunk in. I’ll try not to sound too much like a zealot here, but I actually do now believe that agile is a radical and life-changing approach to software development**. So radical, in fact, that basically nobody does it right***. It’s hard to really, truly internalize it, so I’ll boil it down to the absolute core idea that everything else can be derived from. Ready? Okay. The idea is:

Human beings get stuff wrong a lot.

That’s it! Everything about agile is driven by that fact and what it implies about building software effectively. What kinds of things are we likely to get wrong? Here are a few ideas:

If you knew the answers to these questions, you wouldn’t be reading this blog - you would’ve made several billion dollars running the most successful businesses of all time and investing in the stock market. Nobody can predict the future****. We spend every day doing the best we can amidst a sea of uncertainty. I think we all generally understand this, but agile forces you to confront it, stare it directly in the eyes, every day. That makes people deeply uncomfortable.

Many people probably associate agile with (as I mentioned above) working in small increments and checking in frequently, but it’s important to understand that aspect as a symptom of this underlying idea. Agile doesn’t preach small increments for their own sake - it’s an acknowledgement that what we thought was important two weeks ago might not be important any longer. If there’s a good chance that any decision we make now is actually the wrong decision, the most important question to ask is “What’s the fastest way we can check whether we’re right or wrong?”. Once we check, if the evidence suggests we were right (or it’s inconclusive), we keep going down the same path - presumably we had some prior evidence or intuition that drove us this direction originally. If it looks like we were wrong, we pivot.

This is all incompatible with huge backlogs of tickets and detailed roadmaps. We aren’t rational creatures. We’re subject to a wide variety of biases and fallacies, including things as simple as “I don’t want to deal with explaining to my manager why I’m changing the roadmap, so we aren’t going to pivot”. Sunk cost fallacy is a real thorn in our sides here as well - I put all that work into my estimates and my roadmap and now we’re just going to throw it out? I don’t think so. Beyond that, any time spent planning in detail for work we don’t actually do is completely wasted. If we accept that there’s even a remote chance that we’re going to look back in two weeks and realize that we made the wrong decision, what does that suggest about our chances of being right a month out? Six months? It’s certainly not going to get more likely over longer stretches of time. So if we don’t want to waste our time, the obvious conclusion here is to not spend it figuring out the details until we’re pretty sure that we’re really going to do the work.

In my experience, working without hard roadmaps is more common in smaller organizations where the domain boundaries are more fluid and less common in larger organizations where you may be “dependent” on another team to deliver work to unblock you. I’ve often heard rationalizations in bigger companies for having detailed long term roadmaps that revolve around this - we need to give that other team time to fit in the work around their other priorities, which means we need to plan further out to know what our needs will be. I think this is basically nonsense. This is a paperwork issue that only exists because the other team also has a long term roadmap, which makes it so if we come to them with an ask in three months they have to edit their roadmap instead of being able to pivot to the work we need. This is easily solved by that team also just not having a hard roadmap that far out! All the roadmap does is conceal a deeper, bigger underlying conflict: the other team and my team are not aligned about what the most important thing to work on is. That’s a big problem, and we should probably spend our time figuring that out instead of writing tickets for work we almost certainly won’t do.

All of this makes it harder to change direction quickly, which is the core principle of agile - I wasn’t included in the discussion (I was in elementary school), but I’m pretty sure that’s actually why the original gang called it “agile” when they wrote the Agile Manifesto. All of this is an inevitable effect of having a roadmap with too much detail too far out in the future. They simply aren’t compatible.

*Primarily Scrum, in my experience

**With application to other fields as well

***Trying to avoid the No True Scotsman fallacy here so that I don't end up saying "agile can never fail, it can only be failed" - it’s easy to look at teams that try agile and fail and say “well, you just didn’t do it right”. Agile doesn’t guarantee success and it certainly requires smart and dedicated people to do it well. Conversely, you certainly can succeed with a waterfall project, and some projects are better suited to that style. So it’s not that there aren’t any true scotsmen here - but there are a lot of fake ones.

****Don’t even get me started on psychics