Three Ways Agile has Gone Astray

And How Open Source Can Fix it

In the seventeen years since the Agile Manifesto was drafted, great strides have been made in improving the way software is built. However, not all agile practices have been positive. Agile practitioners have coalesced around several rituals that, though well intentioned, are causing the industry unnecessary harm. The three main offenders are:

1. Sprints
2. Collective Code Ownership
3. Group Estimation

Unwittingly, these practices are obscuring the core principles of agile and promoting poor management. But there is a way out. Open source software utilizes analogous practices that are better tuned to the way software development really works. Agile would do well to model them.

Sprints

Agile processes promote sustainable development. 
The sponsors, developers, and users should be able 
to maintain a constant pace indefinitely.¹

A sprint is a predefined, regularly repeating window (usually between 1–4 weeks) in which preselected tasks must be completed. It is a close cousin to the open source practice of releasing early and often

Open source projects are released at early stages in their development when software is incomplete and buggy. Those with interest in this software are tolerant of the instability and often collaborate on its improvement. As a result, open source software is improved quickly. In The Cathedral and the Bazaar, Eric Raymond describes it this way:

When you start community-building, what you need to be able to present is a plausible promise. Your program doesn’t have to work particularly well. It can be crude, buggy, incomplete, and poorly documented. What it must not fail to do is (a) run, and (b) convince potential co-developers that it can be evolved into something really neat in the foreseeable future.²

This is the great secret of releasing early and often — recruiting the community to help you. Frequent releases maintain collaborator engagement with the product. Raymond compares this style of development to a bazaar, a massively parallel, collaborating group of autonomous individuals. This is different from the corporate practice of picking tasks that must be completed during a sprint, where it is expected that these items be bug free and polished — adhering to a rigorous definition of done. Because sprint goals are always hopelessly underestimated, this practice results in micro-death-marches that repeat continuously without repose. Agile is not leveraging the power of bazaar style development. Rather, this is what Raymond would call cathedral style development (centrally planned, hierarchically directed work), just reduced to shorter release cycles.

To truly leverage a bazaar style of development, you have to release early and often to the right people, your co-collaborators. In most corporate environments, this is not your customer. A buggy, incomplete public release would annoy paying users and likely damage your reputation. You can, however, model an open source environment by encouraging project owners to make frequent internal releases (even when buggy and incomplete) and encourage other engineers to collaborate on these projects with bug fixes and features submitted through patches or pull requests. Better yet, you could open-source your product publicly and follow the model fully by engaging with interested volunteer engineers in the community.

Perhaps the most dangerous aspect of constantly sprinting is the heightened mental state that it generates in software developers — leaving them in an unbroken state of urgency and stress. All the focus is on execution. Ideally, a developer would have regular downtime. Time for reflection or planning. Time to think about coming challenges without daily status reports and burn down charts. Time to prototype and take risks. If these activities are missing, code quality decreases and people burnout.

Collective Code Ownership

Build projects around motivated individuals. 
Give them the environment and support they need, 
and trust them to get the job done.³

It is easy to think of open source development as a collectivist free-for-all where everyone throws code into a shared bucket and magically produces a cohesive, quality product. That is not the case. There is a an important concept of code ownership at play in open source projects where gatekeepers curate and edit contributions they receive. Even though many, thousands even, may contribute to the project, there is an overriding organizing principle that exists in the mind of this gatekeeper. In a very real way, he owns the code. Raymond describes it this way:

The trivial case is that in which the project has a single owner/maintainer. In that case there is no possible conflict. The owner makes all decisions and collects all credit and blame. The only possible conflicts are over succession issues — who gets to be the new owner if the old one disappears or loses interest…

The simplest non-trivial case is when a project has multiple co-maintainers working under a single ‘benevolent dictator’ who owns the project. Custom favors this mode for group projects; it has been shown to work on projects as large as the Linux kernel or Emacs, and solves the “who decides” problem in a way that is not obviously worse than any of the alternatives.⁴

Agile methodologies, such as Scrum or XP, promote sharing code repositories among all engineers. Rightfully, everyone should have read access, but things go wrong when everyone has write permissions. To model open source practices internally you must allow individuals to own the code bases that they work on and be the only ones to check in code directly. Collaborations should be managed through patches or pull requests.

Many benefits spring from this type of strong code ownership. Among them is increased motivation, better design consistency, better code reviews (through pull requests), more autonomy, more hands-on experience designing larger systems, and increased code quality. Agile would do well to shed itself of the collectivist bent it has fallen pray to.

For more on the topic of code ownership see:

Group Estimation

Individuals and interactions over processes and tools⁵

Estimation has always been a contentious and difficult topic in the software engineering community. Some might say the agile movement is largely a response to the unhealthy attempts at estimating and planning that prevailed in the early 90's. Agile proponents rightly condemn the practice of extensive upfront planning. However, the alternative estimation models they have produced are not much better.

In this case, open source development offers no analogous practice for comparison. But in this there is an important lesson. Open source developers have learned that there is no benefit to estimation. This is a hard pill to swallow, especially on the business side of most organizations, but it has proven true time and time again. In fact, no other statement about software engineering comes as close to an absolute law as the following:

The Software Uncertainty Principle

If you know what’s in it, you don’t know when it will ship. If you know when it will ship, you don’t know what’s in it.⁶

Sadly, every day agile programmers sit in meetings dividing poorly understood tasks in to smaller poorly understood tasks. The assumption is that this mental exercise is surfacing and addressing all the unknowns of a project — ignoring the fact that a project’s major issues will remain hidden until coding is underway. At which point, many previous assumptions will be proven faulty and estimation and planning will be invalidated. Exploratory coding and prototyping are much more effective ways to understand the landscape.

In addition to dividing and scheduling tasks in group meetings, agile methodologies employ strange routines for assigning effort values to tasks. Some common approaches are to do “planning poker” using only numbers from the fibonacci sequence, or to rely on the ethereal concept of “story points” instead hours. Some teams have even resorted to using t-shirt sizes or animal names as estimation units. There is no end to the desperate attempts at making software estimation more accurate. Waterfall has been replaced with another set of “tools and processes” to value over “individuals and interactions.”² All of this to avoid the simple reality that software truly is finished when it is finished.

Building metrics around tasks that are divided, estimated and prioritized is driven by a desire to monitor, report and optimize. Software development is stubborn in this respect. It does not yield to the scientific management of assembly lines. The following agile principle captures it best:

Working software is the primary measure of progress.⁷

Because developers are more akin to architects, designers, or artists than factory or construction workers, the best process is the one that defers to their judgement and trusts them to do the best work they can. When managers reach for metrics to motivate developers rather than focusing on working code, they are on unstable ground.

Summary

At the core of agile there is great wisdom. If it can shed the baggage of unhealthy practices that have risen to prominence in recent years and understand that:

The best architectures, requirements, and designs 
emerge from self-organizing teams.⁸

then it will yield software engineering processes that better align with the realities of our discipline and support the well-being of all involved.