How to Build a Large Agile Organization
Okay… consider this scenario. We have a 300 person IT shop responsible for developing control systems that automate large buildings. These systems require front end developers, middleware developers, firmware developers, and hardware engineers. A feature in this system requires us to write code that touches every layer of the overall systems architecture. As it stands right now, the teams are organized around the major subcomponents within the overall solutions framework.
How does a company like this adopt agile? First inclination might be to break up the component teams and create cross-functional feature teams. Each team would have some number of front end developers, middleware developers, firmware developers, and hardware guys… right? The general idea is that any one of these cross functional feature teams would be able to write software for any part of the overall system. Everyone on the team becomes a specializing generalist.
While it’s not impossible to cross train someone who specializes in UI development with someone passionate about firmware (or hardware)… but my guess is that most folks aren’t up for it. My bet is that most organizations don’t have the test coverage to make this kind of experiment safe. I’ve talked with several organizations that have given this a shot, and the change was so disruptive, they couldn’t get anything done. Seriously… nothing done.
I’d like to suggest for a moment, that our goal here isn’t really creating cross- functional feature teams; the goal is to build teams that have everything they need to build an increment of working software. I think that there is a difference between the two. To be agile, we need to to build teams that can stay together over time, become high performing over time, and build trust with their customers over time. They need to build working product.
We tend to assume that in order for this to happen, we need to organize teams around end-to-end features. I agree with this approach when we are talking about small teams and small products. This advice breaks down when we talk about larger teams, and larger… more complicated… more technologically diverse product lines. The agile community needs a more stable, more robust scaling metaphor for discussing these kinds of organizations.
Here is how I have started thinking about this over the past few weeks, and I am starting to think that the language works… let’s start talking about building teams around those things in our organization that are least likely to change.
If we are a small product company, with a single product… the thing that doesn’t change might be a product. If we are a larger product company, with a more complex product offering, I might organize teams around a somewhat static grouping of features. If we are in a really large organization, one with multiple interdependent products, where investments are not level over time… the thing we organize around might be a component or some set of services.
The thing is… it’s not always the flow of features into a product that is often the least transient. Sometimes we can’t build a team with enough specialists… or specializing generalists… to get the job done. When that happens… and at some scale it will happen… we want to start looking for the stuff that doesn’t change. When we find the stuff that doesn’t change, that is where we start building agile teams. That gives us our best shot at keeping teams together.
When you take this approach, you have to keep in mind, that there is no longer any one team that is responsible for end-to-end value delivery. Each team delivers ‘done-done’ work every iteration, but as an overall enterprise, you’ll have to develop the capability to manage the flow of value effectively across each of your various agile capability teams. This is where agile stops giving good guidance and we need something else.
That something else is Lean/Kanban.
So think about this… build agile teams around the persistent objects in your enterprise, whatever they happen to be. Use Scrum or XP or Kanban or whatever other agile method floats your boat at the team level. The teams can become hyper-productive and have as much agile goodness as you can get out of them. To scale all this agile goodness, use a Lean/Kanban approach to manage the flow of value across teams.
So, getting back to my initial question… what’s the most effective agile strategy for our large control systems company?
Organize agile teams around the front end, the middleware, the firmware, and the hardware. Create an enterprise backlog of value features that get broken down into user stories and distributed across teams. Subordinate each team’s backlog to the enterprise backlog, and create a pull system, where new features only get started once the teams have capacity to work on them. Set work in process limits for each team to reduce the amount of work we can get started before we roll back up into a value feature. Invest in constrained capabilities to improve the overall flow of value across the system.
Make sense? Any questions?
Comments (33)
Bob Marshall
Hi Mike,
A significant question indeed, and not just for larger organisations with hundreds of engineers or more.
I have been studying and researching this question for some years now. and have some ideas (for which I have a book in progress).
I concur with your views on flow of value, and kanban as a likely candidate for handling that.
As to thing to organise around, though, I would say that the product (or service) line offers a good choice, irrespective of organisational size. I say this as the product is most often the vehicle by which value flows to the customers.
And as for teams, we are diametrically opposed. I have come to believe that, instead of trying to keep (smallish, agile) teams together, we should be trying to find ways to reduce or eliminate the dysfunctions introduced by projects, and teams. I.E. To find ways in which we can bring people together quickly to work on something, without the usual forming-storming-norming-performing malarkey and jump straight to "performing".
I very much attest to the power of teams, but I believe that power is not a function of teams per se, but comes from more fundamental aspects of people working together. Things like trust, common purpose, shared context, etc. – which I hold can be present outwith the team idea.
I propose one way to work without teams is to form and reform people into groups (of three to five, say) so often (two or three times a week, or even daily), such that folks become accustomed to it, and over time discover ways to make it work better and better.
BTW FlowChain is, in part, predicated on this approach.
HTH
Cheers
Bob
Machiel Groeneveld
One problem with this approach is the stove pipe pattern. To solve that you could add Cross Functional Integration Teams. These are like feature teams, but don't build anything, they perform the integration of what the other teams build. If you rotate people on this team, you keep 'the big picture' an integral part of operations.
A good explanation on this is in Ken Schwaber's; Scrum in the Enterprise.
Mike Cottmeyer
Bob,
I bet that if you and I got together for a beer, and I really had the opportunity to explain the kind of problems I am trying to solve, and the kinds of companies I am trying to solve them for, we would be less diametrically opposed.
From what little I know about your views, I get the sense that you are talking about more of an ideal end state. If I were building a company from scratch, I might not do what I am saying here either. I would probably take a radically different approach to product development, team and organizational structure, and how we all delivered value.
That just isn't what I am taking about this this blog.
What I am doing is trying to help entrenched waterfall organizations who might be open to agile. I am trying to help agile organizations that are open to lean or Kanban. I find that MANY, MANY companies are taking armchair guidance from books and speakers and are really making a mess of things.
They get told that Scrum is simple and in three iterations they can he hyper-productive… it just doesn't usually play out that way.
Thanks for the comment.
Mike Cottmeyer
Machiel,
The Enterprise Scrum book makes certain assumptions about the nature of your organization. It assumes that each component team is nested underneath a single integration team. The integration team is like a Scrum of Scrums that encapsulates the value created by the component teams.
What if you have multiple integration teams that work across multiple component teams? The problem is that your velocity at the team level is not a predictor of velocity at the integration team level. Any one component team can starve any of the integration teams for value.
Furthermore, it becomes nearly impossible to coordinate large scale architectural issues, it becomes almost impossible to make a meet commitments, or have any meaningful capacity to tell the business what they are going to able to deliver when.
I think that for many people this is an academic or theoretically discussion about how an organization 'should' operate, or what 'should' work. Most of my writing is a result of what is possible with many of the organizations I work with. If Ken's approach can work for you, that would be preferable. In many cases Ken's approach will result in the team coming to an absolute standstill.
I tried to address this last year in a screencast for VersionOne. Take a look and let me know what you think.
https://www.leadingagile.com/2009/08/adopting-agile-video.html
Mike
Bob Marshall
Mike,
You're right of course. It was slightly(?) unfair of me to take issue with an approach to scaling Agile to the Enterprise, predicated on a purely Agile perspective. i.e. Trying to make Agile work in the large. :)
And yes, I was talking more about an ideal future state (is there ever an end?).
However, at the risk of confusing further those who may already be finding some challenges coming to terms with Agile, I feel it behoves us to point out that for certain classes of problem (the building of highly-effective development *organisations*, for example), maybe Agile is not the best place to start, nor the most appropriate future state to have in mind?
And I commend your comments to the audience you describe – the transitioning waterfallers and recent adopters of Agile – for whom your words and advice will serve well indeed.
I very much look forward to that beer! :)
Bob
Mike Cottmeyer
Bob,
No worries on the 'unfair' thing… it's a blog for God's sake ;-) I'll call you next time I am in London for the beer. I need to find a reason to head to the UK for a bit.
Mike
David
Taking a somewhat disruptive approach, you could organize teams into the Problem Group & the Solution Group. Each one could be made up of cross functional teams, and perhaps it would alleviate some of your concerns listed above.
Most of the leanstartup guys are preaching it, but it may be more suited for the startups.
Mike Cottmeyer
David, do you have a link to a writeup or something you could share. I am not familiar with the approach.
stanimir dochev
Hi Mike,
I am part of a group that is treading along the agile way. Your ideas help me get over some of the hurdles and give answers to pending questions. So, thank you ;- )))
Last week I watched a small movie about one of the Porsche's factories and how they implement the lean principles. They separate the whole production into groups and let each group act as a separate scrum team. For example, the production is separated into logistics in the factory and assembly. Each of the logistics and assembly units are trying to optimize. The whole process is directed by a central unit which in turn has its scrum sprints. It sounds like a waterfall on macro level and agile on micro level.
My experiences so far in the software industry point to something similar. The efforts of having one cross-functional feature team that can deliver a complex product, sooner or later result in a bunch of small scrum teams, working as you call them around the "persistent objects in the enterprise". Each of them acts as a stakeholder for the others and "pulls" the feature it needs.
Maybe this is the first and less painful step towards a better implementation, but I am still to experience it. So your proposal sounds like the best possible solution at the moment.
Greetings,
Stanimir
David
Mike,
Most of the material can be found online via Steve Blank, Eric Ries & Dave McClure, however slide 4 of this deck touches on it: http://www.slideshare.net/startuplessonslearned/2010-01-11-customer-discovery-crash-course
As I said, it is mostly geared towards startups but could be leveraged in the enterprise.
Perhaps I'm going on a tangent with this, but large organizations could very well defend market share or dive into a new market using these techniques. That is if they would let go of traditional silos and use a cross functional, iterative approach for both product and development…
Pawel Brodzinski
Makes sense although it can be pretty difficult to implement. The problem which I see is exactly the same which appears during scaling Kanban up. At some point you need a very good product owner/product manager who is able to deal with the whole product.
Yes, they can work on pretty general level but still they should understand what's happening with the product(s).
If you think about situation with multiple products the focus and responsibility goes to some general manager or someone who is able to coordinate work between different projects/products.
In short wherever you have more than one product manager and shared pool of builders you'll always see conflicts between products. It takes a strong leader to solve them. And that's exactly what I'd focus on in this approach.
Vasco Duarte
It is hard to understand from this post what you are trying to scale.
Are you scaling?
– "management of teams"
– "management of work"
– "management of requirements"
– "management of value"
And what do you consider to be the obstacles to whatever you try to scale with the feature-team approach?
There are existing models for "scaling project management through requirements". Here's one view:
http://bit.ly/c6JNX5
I'd be interested in continuing this dialogue, I think there's something we need to develop further in this area.
Mike Cottmeyer
Stanimir,
Thanks for the comment, I hope the approach works for you. I am starting to think that for large complex organization, this isn't just a less painful model, it might be the only rational model.
Let me know how things go for you.
Mike
Mike Cottmeyer
Pawel,
Totally agree. You need not just a great PO for each product, you need a great PO for each static component team. You also need great agile portfolio management and some degree of architectural and product visioning early in the project. You also need your organization to start breaking projects into smaller chunks so you can increase the flow through the overall system. Use the same INVEST model we use for stories to scope projects or large value features.
Anyway… this wasn't my final post around this topic, I expect to unpack this more fully over the next few months.
Mike
Mike Cottmeyer
Hi Vasco,
In some sense, I am trying to scale all of the above. At the end of the day, we are trying to get the overall system to deliver value faster. In many cases, it doesn't do me much better to get good with a single team if the other teams are not operating with each other in a coordinated fashion.
There are many cases where the feature team model would be the preferable approach. There are other cases where the component team model is the preferable approach. When someone tells me we have to have feature teams at all costs, I ask them to do a thought exercise to come up with some environment where a feature team would not work… what would they do then?
Even if you can encapsulate all of development into a feature team, what do you do with sales and marketing, what do you do with downstream support? They are part of the value stream too… ideally we need to have a way to manage them as well. Should they be part of the dev team? No. This approach works for managing non-IT groups too.
I guarantee you, that there is some example in your environment, I can come up with, where the feature team breaks down. Again, what do we do then? I am coming to the opinion that the 'feature team at all costs' mantra is really harmful to many agile implementations.
Mike
Mike Cottmeyer
One more thing Vasco… usually when I find someone saying they are a 'feature team' in a large organization… they are really a 'component team' that has chosen not to pay much attention to the other teams around them. This leads to suboptimization at the enterprise level.
r
I agree with Vasco that there are more questions that options from this post.
What makes this 300 person IT shop any different than the 100s or 1000s of other 300+ person IT shops? Are you trying to say there is something fundamentally different due to the firmware/hardware aspect compared to other corporate/enterprise IT shops, or is it solely based on size?
And what is your "in" to this environment — is it a top down push of agile from management or is a grassroots effort from within a department or team?
If an aspect of agile is pushing down and distributing decision making responsibility across the spectrum (and being held accountable for those decisions), how does reinforcing stove-piped isolated systems development (but "agile" within the pipe) help?
How do you differentiate between building around "things that are least likely to change" and the need to actually be disruptive because status quo isn't working?
Mike Cottmeyer
R,
Let me see if I can address some of your questions w/o writing an entire blog post ;-)
Yes… I am saying that size and the degree of required specialization within the systems architecture matter when you talk about scaling agile in a large diverse organization.
What makes them different are things like being spread out over the world, products being made up of other products, multiple product owners at play, and yes… hardware guys don't do software and software guys don't do hardware.
My 'in' into this environment is at a senior enough level to have an impact across the entire delivery stream. Most agile that starts with the team, in a large enterprise, will end up as a suboptimization with everyone wondering why we can't actually get product out the door any faster. I have another strategy if I am starting at the team level and working my way out… you can be successful no matter what direction you are coming from.
I agree that there is an aspect to agile about pushing down decision making, but that does not mean that everyone on the team gets to do whatever they want. There are constraints that come down from the larger organization and feedback loops that go back up. This is required for coordinated decision making across teams.
I would disagree that this reinforces stovepipes. Like I said to Vasco, in many organizations… my feature team is your stovepipe. It all depends on perspective. How many agile teams do you really know that OWN the product end to end? Not many. If you think they do… ask if they own marketing…. do the own sales.. do they own support… do they own packing and integration and services after the sale? All of that is part of the value stream.
I am all for disruptive change, as long as it is change with a purpose. As long as it is change that will work.
So let me flip the question back to you… tell me how you would scale the organization I am talking about here? Remember… 300 people… at least 4 very different and distinct skillsets… everyone of them is required to bring the product to market as fast as the organization wants… multiple products supported across a common architecture???
30 cross functional feature teams???? Try again… or try to convince me. Your call ;-)
r
Let me pull out my logical fallacy cheat-sheet so I can identify them properly for you. :)
How can I answer "30 cross functional feature teams" when I never mentioned the concept, even indirectly? I don't need to try an "convince you" seeing all I did was ask a few questions. So put your little biases in a shoe box a shove it under your bed for a few minutes.
You must know, even when writing the original blog post, that there is no one answer to your question. Books have been written on the topic such as "Agile Software Development in the Large" by Jutta Eckstein or "Scaling Software Agility: Best Practices for Large Enterprises" by Dean Leffingwell. Not that I'm endorsing either as answers — if a book could solve an enterprise's agile transformation you wouldn't have a job. But, it certainly isn't something that can be answered in a blog post response.
So, all I have for context is the words on this page… which really ain't much considering the size of the problem. (Yes, "ain't.") And that's all I was trying to get at with the questions too, there is no cookie cutter formula, there is no agile formula. Right?
But in light of all that, I think there is a clue in your last response. I personally don't think it is possible to 'shove' agile down from the top. Your "in", as you state, is at a senior level, but that doesn't mean agile has to be shoved down. You're in an ideal position because a lot of the problems with bottom up transformation is management push back.
You said "I have another strategy if I am starting at the team level and working my way out… you can be successful no matter what direction you are coming from" — I'd say that would be a starting point. If you have a strategy and that you can be successful at it then is that not a place to work from?
Daryl K
Hi Mike,
I read the post and the first few comments.
I think you're off base. You're trying to allow the organization to stay organized the way it is because it is simpler and it is so hard to turn a big ship.
But end-to-end early is the only way to show actual progress. If you build some internal component that has no inherent business value, as a product owner, I see ZERO progress. It seems like progress to a techie, but from the business-side, zilch. So now we've gone an iteration and you've gotten no feedback from me.
The purpose of those tight iterations, the central purpose, is to show progress to the business and get feedback.
I can't say that I've worked with a team of 300, but I have coached a team of fifty-five. We worked cross-functional on weekly iterations.
When you say a cross-functional team, you jump to "specializing generalists." To me, those are two concepts. A cross-functional team could be a team of specialists who do not know multiple layers. A UI specialist, a business rules specialist, a data layer specialist. They don't know the other layers and they don't intend to learn. But they are cross-functional because they work closely with people from other layers, not in their own layer cocoon. This way, we can build end-to-end early as a team.
In my early experience with projects, it became very obvious that big projects always failed and small projects often succeeded. The only way I can think to tackle a big project is to strip it down into little projects. And the best way to take those strips is end-to-end, because it is the only way I can show business value and get business feedback.
You asked me to read it…
Larry Maccherone
Mike,
I'm late to this discussion so I won't jump in to the discussion on the "art of the possible" about creating cross-functional teams when it's hard. However, I do want to comment on your core ideas. I want to first say that I admire your bravery for bringing up this valid alternative to the standard agile recommendation on having cross-functional teams. I recently started rolling out a scrum-based process to a large product development effort that is 80% mechanical and electronics and only 20% software. In a situation like that, you can't carve up the product into feature teams. You have to leave the discipline teams in tact.
Your post starts by recognizing this difficulty but then you float this principle that teams should organize around the thing that is least likely to change. My first response is, "interesting, let me test that with my own experience." But in my experience, I only run into the same three cases that you mention and it's hard for me to say that your principle helps much in those situations. I'm all for going back to core principles when making decisions that are outside of the known patterns but my experience only has those three patterns so why not just capture the characteristics of those three patterns and apply them? Why do I need a principle?
So, what are some situations that folks have seen where you could more easily identify the thing that is least likely to change, then you could say which of the three alternatives is a "better fit" for the situation?
If we can't find a lot of meat for the above question, then I would love to engage in a discussion around the conditions that indicate which of the three alternatives is a better fit. I think this is an important issue that folks have been afraid to bring up… but where pragmatists have been silently deviating from the standard recommendation when… well… pragmatic.
Mike Cottmeyer
Daryl,
Thanks for the comment. I agree with most of what you said. Is there any case where you could see a 'cross functional feature team' not working? If there was such a case, what would you do to mange the work to make sure the business got value? That is the case I am talking about here.
I agree that making smaller projects is better. Is there a risk that we make the project so small to get our 'cross functional feature team' that what is valuable to the team isn't any longer valuable to the enterprise? Is there a risk that we end with a 'cross functional feature team' that is really just a component team within the larger enterprise? Because really, unless the team is responsible from concept to cash, which most are not, there is an element that has to be managed across teams.
Wishing all projects could be broken down such that they are meaningful to a team of 6-8 and still relevant to the business doesn't make it so. Good conversation, thanks for the comment!
Mike
Mike
Mike Cottmeyer
Larry,
Thanks for the comment. I started using the language of 'looking for things that don't change' not so much a rule, but as a way of starting the conversation. I used to consult for VersionOne. When I was helping folks setup their tree structures in the app, I would always look for what was consistent and what was transient in each organization, and model the persistent things with one set of objects, and the transient things with another set of objects.
Trying to build teams around things that are constantly changing doesn't lend itself to keeping teams together. You can make the argument that any team can solve any problem, but that doesn't always hold true. To me, the core of agile is teams. If we can't start there, we can't start. I would rather build teams and manage the flow of value across teams, than try to construct and deconstruct them based on the needs of the initiative.
Anyway, doubt I addressed your question. Maybe another blog post later ;-)
Larry Maccherone
Mike,
Actually, it did sorta answer my question because now that I better understand what you were getting at, I can come up with examples from my own experience that fit.
So your principle didn't help much but your comment above reminds me that you are trying to optimize for keeping the teams together. You may have said that in the original post but it got lost in my reading. You are saying that it's more important to keep the teams together than to get to fully cross-functional teams. So maybe your next blog post should be a discussion of what you trade-off by not being cross-functional.
Thanks,
Larry
joshilewis
I have two concerns with this approach: You talk about "Each team delivers 'done-done' work every iteration". My contention is that design is a completely non-sequential process, and cannot necessarily be done without all parties present, including both 'different level people' (i.e. middleware, UI etc) as well as 'different activity people' (i.e. developers, analysts and testers). I therefore don't really believe in non-cross-functional teams being able to deliver to 'done done'.
Stemming from that, I believe that organising people around activity (as opposed to deliverable features), where we try leverage according to 'testing' or 'development' leads to availability resource bottlenecks. If as a developer I need clarification from an analyst (or a tester from a developer, or a UI developer from a middleware worker), the longer I need to wait to talk to that person, the more the team suffers. For me, the aim of a cross-functional team is to make all the necessary people as available as possible.
There was a little discussion of this on the Kanbandev mailing list a few weeks ago.
Amit
Hi, your blog is really nice & wonderful, while reading it I truly like it.
mba india
Stan Yanakiev, PMP
Hi Mike. This is an interesting post as scaling Agile up is really a crucial issue. But how is what you describe different from traditional matrix organization and herarchical structures? There are differences but niot that significant. Boundaries are blurring to me.
Stan…
Not sure quite how to answer your question. Traditional orgs group people by role specialty. I am suggesting that we create cross-functional teams aligned with business objects like products, services, or maybe even business processes. We have to have control mechanisms for coordinating the output of multiple teams. It’s naive to think that the entire value stream will be contained within a single scrum team. Make sense? Answer your question?
Stan Yanakiev, PMP
Mike:
The approach you suggest will probably work. I am trying to understand what is the novelty in it. Some agilists claim standard management methods and structures are totally obsolete and agile organization is possible without them. I haven’t seen such a model yet, even a theoretical one. On the other hand, cross-functional teams also exist today when a matrix organization launches a project and people come from different departments. Probably there are departments that have people with different skill set too. But if it is some specialized department, say accounting, cross-functional teams wouldn’t be that good idea I think.
I think you might be missing a nuance here. I’m talking about creating a cross-functional teams of people that stay together over long periods of time that are organized around a business object within the enterprise. These people can have a functional manager, and therefore be ‘matrixed’ into the team, but I want the strong side of the matrix to be the team, not the functional group. I am not talking about project teams, I am talking about persistent delivery teams. It requires an understanding of the organizations business architecture, an understanding of the flow of value across delivery teams, and a mechanism to manage the flow of value to deliver end-to-end business outcomes.
And for the record, I don’t claim anything on this blog to be insightful or novel… I just write about my ideas, opinions, and observations in hope that they might help someone else who is struggling with the same issues ;-) Hope this reply helped!
Stan Yanakiev
I see. Organizing persistent cross-functional delivery teams teams around “business objects” in a company is a way to scale up agile. The teams may not stay together over long periods of time though. When the business objects change, the teams will have to be reconfigured and the company structure will change. For instance a company may shift from development of desktop applications to servers. Changing the teams, management and the structure may be good as it provides flexibility but there may be some challenges as well.