A few months ago I co-authored a white paper (with V. Lee Henson) on the…
As usual… I’ve gotten a little sidetracked on my Agile 101, 102… series of posts. I’ve got the 201 post half finished, but haven’t been able to spend the time getting over the hump. That said, I want to divert just a little and talk about agile in general, explore some of what it is, and a little about where it’s going. I think we have created a bunch of confusion in the marketplace. I am under no illusions that this post will be the definitive explanation, but I spend a TON of time talking to people and level setting before I can have any kind of intelligent conversation on agile, so I want to explore some of the points that I think resonate.
Can We Define Agile?
First of all… back in my early, early days of community involvement at V1… I was really wresting with this untangling language around agile. I used to break the conversation into a few key areas: project management, product management, leadership thinking, and technical practices. Today, I find myself taking a slightly different angle. I tend to see agile discussed as a cultural phenomenon, sometimes as a set of practices, and sometimes as a way of structuring and governing how your company works. More often than not, agile is setup as a counterpoint to some of the most ridiculous examples of bad management we can come up with in an industry.
From my point of view, agile is all of these things collectively and none of these things individually. Agile is about having a rational system of work in place where people are engaged and can thrive, but one that produces the business outcomes that our stakeholders are counting on. When we get too myopically focused on one aspect over others, it is easy to start talking in platitudes and one-size-fits-all adoption approaches. Every organization I’ve ever worked with has been vastly different and, while certain patterns apply, solutions are always unique to the people that are impacted by the change.
The first line of the Agile Manifesto says that we are uncovering better ways of developing software by doing it and helping others do it. How we implement agile is supposed to change over time, and how it changes should be guided by the values and principles in the Manifesto. The challenge is that the values and principles are supposed to guide the practices and structures we choose to implement. The values and principles don’t live by themselves or in a vacuum. To successfully implement agile, we have to have a system of delivery and supporting practices which enable the values and principles to be lived out.
The Problem With Big Companies
I’ve been focused on what I call the ‘Big Company Problem’ since I left CheckFree more than 7 years ago. Back in the day, we were building complex systems of systems for the financial services industry. Products of products. Multiple overlapping and intersecting value chains. Heavy traditional governance and pockets of agile scattered throughout the organization. This was an environment with specialized business domain expertise, multiple diverse technology stacks, an organization that demanded 5 9’s reliability where downtime immediately impacted revenue and incurred penalties. How does one begin the process of agile adoption in any meaningful way in an environment like this?
Many of us suggest that ‘big and complicated’ is the problem and that we need to be ‘small and simple’. I agree… but, these companies ARE big and complicated so the question becomes HOW to help them become small and simple. Just saying BIG is bad and you should be SMALL doesn’t help. How do we get there? What is the path from A to B? Is adopting an agile value system enough? In the presence of the right value system, will the right delivery systems and practices emerge? In the presence of the right practices, can I impact the end-to-end system of delivery and make the cultural changes I need to make?
The problem with the culture first mindset is that there is practically no way to change enough hearts and minds to get everyone on board fast enough to produce results in the timeframes we are working against. Even if I could get everyone to adopt an agile mindset, we have real governance issues and audit requirements that often cannot be changed overnight. We have tightly coupled technology stacks, tightly coupled product requirements, imbalances in capacity and demand, and bottlenecks all over the place. Solving the problem is a matter of redesigning the system of delivery. Not everyone is an expert in designing systems, and even those that are don’t agree on the best path forward.
There are clearly some places where a culture first transformation can work. There are definitely some where it won’t. As an industry, we have to have an answer for when the scale and complexity are such that self-organization around the problem isn’t a viable approach. Again, the right company, the right business problem, the right underlying technology, and the right group of really smart people can get a ton of stuff done given the right degrees of freedom. There are some environments, though, where this is recipe for chaos. I personally think it is irresponsible to suggest that cultural transformation is going to fix this. We cannot always assume that if enough people ‘get it’ agile will begin to take hold.
Here is another one that I believe is driven by our long history of thinking of agile as a set of processes. Most top down organizations are accustomed to using process to pull together disparate working groups and coordinating activities across them to create larger deliverables or achieve broader organizational objectives. In many of these cases, the underlying organization, and maybe even the underlying organizational culture, doesn’t matter all that much. All we need to do is document a process that people can follow and, provided they actually follow it, we’ll get the outcomes that we desire. People assume that agile as a process will behave similarly.
Agile is different. You can’t take a set of functionally siloed teams, operating under heavy management control, traditional phase gate governance, fixed time, cost, and scope constraints, and tell them to do whatever process they want. You can’t tell them that agile is okay, but then hold them accountable to live in the same organization, under the same rules, and under the same constraints as a waterfall organization. The agile process in and of itself doesn’t make any rational sense outside the context of a team based organizational model, an agile governance framework, and a mindset that allows for some adaptation to the unknown. It doesn’t work.
Practices alone will never solve this problem. All we end up with is a bastardized form of agile where people go through the motions, but where agile doesn’t live up to the value that was promised. In order to do agile well, you have to have teams, you have to have backlog, and you have to have the ability to produce a working tested increment of product at the end of each iteration. This involves rethinking how you go about forming teams, how teams work together, and how value is coordinated. In the presence of teams and practices, I believe that the right thinking can emerge.
I’m a big believer that agile culture and agile practices are essential for long term sustainable organizational agility. That said, I don’t think you can train your way into it and I don’t think you can believe your way into it. I think you have to start with a team based organizational design where the flow of value is governed across teams using an adaptive governance model, based on lean/kanban principles, where WIP is limited, capacity and demand are balanced, bottlenecks are identified and dealt with, and management is invested in improving the overall system of delivery. Only within this kind of organization can an agile culture emerge and agile practices thrive.
I also don’t think that people intuitively understand these kinds of systems and I don’t think that most organizations will self-organize their way into them. Most people tend to think about optimizing what they can control and getting better only at what they are responsible for delivering. People are often self-interested and will consciously or sub-consciously advocate for system designs that are in their best interests. In the long run… forming teams, decoupling dependencies, reducing complexity, and creating an ecosystem where small, independent teams can self organize within their boundaries is the only model I believe will work.
So, I agree that we need to be small and simple, but getting to small and simple… much of the time… is a process of forming teams around business problems and technology, progressively decoupling them from each other, reducing dependencies, and dealing with bottlenecks. There are real issues facing large enterprises, there is a ton of momentum around the old way of doing things, real organizational and technological constraints, and just telling people to self-organize around a new system of delivery… one that they may or may not collectively understand… is often a recipe for disaster.
Structure then Practices then Culture
We do though have a bit of a chicken and the egg problem here. How do I get permission to start an agile transformation if I don’t have a leader with the right mindset to give it a try? I’d suggest that you at least have to have a leader that recognizes there is a problem and is open to alternative ways of solving it. I’d suggest you begin by focusing on your business goals, articulating a strategy to create a team based organizational model, and model based on iterative and incremental delivery principles, one that uses agile and lean methodologies for delivery and governance, but that operates within the operational and cultural constraints of the existing organization and it’s policies.
You teach this organization agile technical practices and management practices at the team level and adaptive lean governance practices at the program and portfolio level. You teach the organization how to deliver with reliability and predictability within that framework as you build trust in the new way of working. As we learn the capability of the new model and build trust with the organization, we create the opportunity to influence hearts and minds and create the environment where people feel safe to take chances and absorb risk with new ways of thinking that challenge their old ways of thinking.
The Role of Leadership
These kinds of changes have to be led with top down intent, but with a bottom up implementation. People have to be engaged and do ultimately have to be bought in. That said, buy-in will only come when people realize that the new system is working and the new system rewards the new behaviors. Once we have the rational team based system of delivery, new practices that support operating the new model, and a new mindset that enables us to unleash the power of the new system of work… we can start really improving as an organization and start tapping into the promise we’ve been reading about with agile all these years.
IMO… it’s not so important to talk about agile as structure, practices, or culture… it really is all three. It doesn’t matter if we are talking about agile as a set of management practices, an approach to product development, a leadership framework, or an approach to software craftsmanship. It is all of those things as well. But when it’s all those things living in an integrated system of delivery, where all the parts work together, when they are no longer in competition, where we really start to see the value. Things have to work as an integrated whole. It’s the working together as an integrated whole that is going to make this take off. Not just some team level agile practices or pockets of agile mindset in a vacuum.
Maybe this is all obvious, but there is a lot of debate about where to start with your agile transformation and what’s important to focus on first. If not debate, then different messages and approach in the marketplace around what works. I think that many folks work in broken organizations and can’t see a path forward… because without the support and direction of your senior leaders, nothing I’m talking about is even possible. As an industry though, once we get the messaging right, you are going to see more and more agile adoptions led from the C-suite, not in silly Dilbertesque caricatures, with informed intent around how to build organizations with a systems based perspective.
In most large organizations, bottom up is not an option without leadership acting from the top.