Learning to be Agile!

Principles behind the Agile Manifesto

Experience is learning the hard way that something won’t work, is harder than it looks, or will come back to haunt you.

Learning from history is where you find that something didn’t work, was harder than it looked, or came back to haunt someone... and then you go ahead and do it again.

The well-known quote that “history does not repeat itself, but it rhymes” is, of course, attributed to Mark Twain. Quotes like this can be very hard to trace, as the ideas may be widely expressed in different ways, in different languages, and at different times. It also frequently happens that people “retcon” something similar that the person actually said.

In this case, Quote Investigator indicates that Twain was first credited with the saying in 1970, but that no evidence could be found of him actually writing it, though he did write something similar: “History never repeats itself, but the Kaleidoscopic combinations of the pictured present often seem to be constructed out of the broken fragments of antique legends”. More poetic, perhaps, but less pithy.

The idea of history going in cycles is a fairly common one, but rather than a back-and-forth or boom-vs-bust image, I tend to think of progress as more of a helix, or coil-shape.

An illustration of this is the history of computers. In the earliest days, processing was done by the computer, and the console was an integral part of it. Later, terminals allowed multiple people access, using “dumb” terminals. Then, in the early days of personal computers, the computer and terminal were combined, and processing was “back” on the computer.

Did we just come full-circle?

I would say no. Instead of a single, centrally-located machine, with one or more operators, we now had multiple, distributed machines, each with a single operator. (And, of course, we still had larger, multi-user computers)

Then we started building client-server networks, in which we had “smart” clients using server-side resources. Over time, some software moved to the server (or never left), and we accessed it via terminal emulators – so, the smart client ran an emulator of a dumb terminal to access the server.

How about this time?

Again, no. This time, we had a single, centrally-located machine, but the terminal emulators were simply one program among many being run on the distributed computers.

While a student of history can usually see what we’re doing and can make some pretty good guesses as to where things might go wrong, people usually insist on making their own mistakes, so we usually don’t progress as much as we might hope.

I recently listened to Cybersecurity Today: Month in Review – Microsoft Patch Fails, Fortinet Issues, and AI Risks, in which David Shipley described his company (Beauceron Security) as reviewing their software development life-cycle policies, and moving away from the Agile Manifesto.

Now, before the project managers pick up their torches and pitchforks, I should add some fuel, by noting that “Agile” can mean pretty much whatever you want it to mean. It’s frequently used as a catch-all term for a whole collection of different tools and practices which are (sometimes loosely) tied to the concepts expressed in the Agile Manifesto.

This document was the product of a meeting in 2001, where a group of software developers gathered to discuss “lightweight” development methods. This was a response to growing frustration with the “waterfall” model, which was the first documented SDLC (Software Development Life Cycle) methodology.

The original waterfall model was mainly theoretical in nature, and even the earliest descriptions of it note that the approach would vary by the nature of the project, and was not exhaustive. As so often happens, though, many organizations treated it as “the way” to run projects, rather than a starting-point for discussion and further work.

This, of course, led to poor adoption, project failure, budget overruns, and resentment, especially as the rate of change in the industry increased and the gap between business and technical teams grew. To address this, many new approaches were developed and promoted, with varying degrees of success, until that meeting in 2001.

The Agile Manifesto helped unite various approaches under a single umbrella, so when a company adopted “Agile”, they could mean Scrum, Kanban, Scrumban (these are real approaches), or others. In many cases, large organizations developed their own hybrid approaches, but there is a tendency for organizations to prefer a single approach for all projects, usually out of a desire for consistency.

I think the real problem is that, while there is enormous value in business and technical teams working closely together, this frequently (usually?) means that the business prioritizes work in such a way that things like security and quality end up being pushed aside to be addressed later. And, since the principles laid out in the Agile Manifesto include things like “working software is the primary measure of progress”, business teams often “win” in the push-pull of prioritizing work.

But, the Agile evangelist will likely say, “working software” always includes security and quality checking, and the business teams understand this. To this, I laugh uproariously and ask if they have ever worked in software development.

Actually, while I understand and support what Mr Shipley is trying to do, I don’t think it’s necessary to move away from the Agile Manifesto to ensure quality and security. Instead, an organization needs to clearly establish definitions and baseline quality standards. (That said, it may be that Agile is not the optimal approach for a given organization or project, but I believe that’s separate from quality and security.)

Many Agile teams work to build a Minimum Viable Product (MVP), then add on additional elements over time. Often, this MVP is little more than a screenshot and “stubs” of functions and features, which are added to over time to build out the product. Then, later in the project, the technical team is under enormous pressure to deploy the “finished” product, and things like quality and security are not prioritized.

While this is what often happened, I would say that this is not consistent with Agile principles. Instead, I would argue that insufficiently-tested and insecure code is not “working software”, and that the quality and security are simply baseline requirements.

Unfortunately, addressing this at the project level is often ineffective, as most project managers or technology teams do not have the authority to push back effectively. This is why organizational processes and automation are probably the best approach.

Traditionally, technology teams would build in a development environment, and then push updates to one or more test environments before they are eventually deployed to the production environment. Nowadays, this is usually automated to a significant degree, and driven by pre-defined workflows.

If the developer is simply unable to deploy code without a validated library of test cases, and unable to promote code to another environment without testing, and the project manager is unable to release a build without a security signoff, these things will eventually be simply the way things work, instead of obstacles which can be overcome or worked around.

All of this, in my opinion, is entirely consistent with the Agile Manifesto, and can be an effective approach to development. Is it easy to set up? No, but it’s almost certainly easier to establish some additional workflow steps and checks in your already semi-automated processes than to overhaul how business and technical teams work together.

The part that is hardest, but most necessary, is understanding what you want to do, figuring out how to do it, and defining effective processes. Deciding whether or not to “go Agile” does not change that fact.

Cheers!