Balance the System First

WRITTEN BY Mike Cottmeyer

Let’s say you are a company that builds an enterprise class software product, one that in general is pretty standard, but always seems to require some degree of customization… or maybe a new feature (or two) to get the big deal closed. Your product is much better than the competition, and has always been extremely high quality, and sales skyrocket.

Over the years, sales have continued to grow, and along with those sales, the size and the complexity of the product has grown along with it. Over time, the rate at which you can add new features into the product has gone down, even as the size of your development organization has doubled over that same period of time. To make matters worse, product quality is deteriorating, and you are on the verge of loosing a few long time customers.

The pressure to deliver new features gets more and more intense, while the quality of the product continues to decline, along with the rate of new releases.  It’s easy to think this is an engineering problem, but let’s look at the system a little more holistically, and explore how the whole system from sales to support contributes to this problem… and if we are going to fix it… contributes to the solution.

The Impact of Sales on Product Management

The whole process around selling puts pressure on Product Management in a number of ways. First, Product Managers are pulled into the sales process to help close the deals.  Over time, they are less available to the teams. Second, there are more new requirements that have to be written, and fed to the development organization, to keep all those new features moving.  The net effect is that we have more requirements, less time to write them, and less time to spend with the developers to clarify them.

The Impact of Product Management on Development

We now have Product Managers that aren’t as available as they used to be, and the requirements they generate are higher level than they used to be, and there are more of them (requirements) than ever before.  Because the requirements are not sufficiently broken down, and not sufficiently understood, and the Product Managers are not available.  We go into development with a significant set of unknowns that are left to the developers to work out as they go.

Furthermore, the sheer volume of requirements is more than can be built within the timeframes that Sales has sold.  As Development slows down to analyze more and more requirements, less features actually get delivered, and more often than not, features get re-prioritized constantly to accommodate angry customers and mitigate slipped schedules, which results a ton of waste and re-work.  All the while, sales continues to sell, and the backlog continues to grow.

The Impact of Development on QA

Given fixed dates, limited resources, and apparently inflexible requirements… developers will do what they can to show as much progress as possible, often sacrificing quality to make a dent in the backlog.  Who needs tested code anyway, that’s what we have QA for, right?  The sheer volume of software that builds-up creates an ever growing backlog behind QA, and QA never can catch up.

To make matters worse, poor quality software has to go back through the development/QA cycle due to the vast number of problems QA finds in the code. Because (as we know) quality isn’t just about defects, it’s also about the maintainability of the code… as technical debt grows, the code gets harder and harder to work with, and harder and harder to test.  The defects that are found are difficult to fix, and fixing them often result in introducing even more bugs into the system.

The Impact of QA on Support and Sustaining Engineering

Because QA was pressured to quickly test the software, defects are often missed and passed into production. These issues result in more trouble tickets, calls to support, defects that have to be logged, tracked, and managed… and at some point, all that defective code has to be fixed. Because the sheer number of issues, and the fact the code is full of technical debt, Sustaining Engineering hits capacity pretty quickly.

The overflow from Sustaining Engineering either feeds back into Core Development, or back into Sales as issues get escalated, which ultimately feeds back Product Management, which feeds it to Development, which feeds it to QA and the cycle feeds itself in a never ending spiral.  Eventually the entire organizations ability to create new features and deliver working software grinds to a halt.

What do you do?

Okay… I think it’s clear that we should never have gotten ourselves into this mess… but now that we are here, what do we do about it?

  • Should I solve the problem by hiring? If so, what role(s) do I hire?
  • Should I solve the problem with training? If so, who do I train?
  • Should I use better engineering practices?  If so, which ones?
  • Should I use more automation?  If so, where do I start?
  • Should I solve the problem with process? If so, what process?
  • Should I solve the problem with pressure? What’s the impact of putting more pressure on this system?
  • Should I solve the problem with better management? Will efficiency gains make this system behave more predictably?
  • Should I stop selling new product? Should I only sell things that are off the shelf?

Non-Trivial Problems

This clearly falls into the category of non-trivial problems… and like most non-trivial problems they have no easy answers.  The most important thing to realize though, is that people have careers wrapped up in the answers to this question.  Admitting you are in trouble is hard and taking the steps to fix the problems are even harder… while the destination is clear, the path from A to B is going to be difficult.

The core of the issue is that the demand for new software is far greater that our ability to produce new software. Even though our development staff has doubled, our ability to create new features is now reduced by the amount of time we spend defining the product, the time we spend working on legacy defects, the time we spend working with hard to manage software, the time we spend testing software, and the time we spend actually paying down our technical debt.

Getting Real

Our job is to get real about that… we have to bring our capacity into alignment with the demands of the organization.  Sometimes that means we have to reduce demand… sometimes that means we have to increase our ability to produce quality working, tested software.  Bringing new people into a broken system usually just makes the broken system worse. I recommend starting to throttle back demand until we can get the system stabilized and balanced.

That means we might need to stop making commitments we know can’t be delivered, it might mean that Product Managers should work more closely with the teams, or it might mean that we need to hire some business analysts.  It might mean that we need to help QA by producing less crappy software and having developers focus on better unit and integration testing.  Developers can even help with manual testing in many cases.  We can reduce the load on support by delivering better software in the first place and fixing the stuff we have in production before we build more.

Maybe

Maybe people in our organization need to move around.  Maybe some people need to find a different place to work. Maybe we need a more disciplined process, or execute the process we have in place more efficiently and more effectively.  Chances are we need a little bit of all these things… the trick is realizing that we don’t have a point problem that requires a point solution, we have a system problem and we need a systematic solution.  Fixing one part of the system will not generally make anything all that much better… sometimes it makes it worse.

Once we have the system in balance, now we can start thinking about where to hire or how to spend money… in a way that improves the entire system.  Bringing the system into balance has to be the place we start if we want a long term solution that will actually fix the problem is a sustainable way.

leave a comment

Leave a comment

Your email address will not be published. Required fields are marked *

11 comments on “Balance the System First”

  1. sachin kundu

    Ah classic case for remembering Theory of constraints. As you put it “throttling back demand until we can get the system stabilized and balanced”

    If sales put more data than can be handled in the pipeline it is going to create WIP and consequent waste. First solution would be to cut back sales(as unintuitive as it may sound) and then see where the bottleneck moves.

    Reply
  2. Agile Scout

    Couldn’t have said it better. With LARGE enterprises, this is exactly the case. What would you say would be the FIRST way to start balancing the equation?

    Reply
  3. Mike J. Berry

    As applications mature over time these types of dynamics start to take over a manageable environment, and the organization starts to feel the weight of the market demands. I like the comments made about having to pay back technical debt accrued by irresponsible development early on. Architecture reviews by experienced architects can eliminate a lot of technical debt. Good OOP practices, such as S.O.L.I.D., etc. can also provide a safe platform for effective scaling while minimizing technical debt. You should insist on this in your environment. Even for the smallest projects.

    One angle not mentioned is the need for an organization to be able to prioritize desired functionality appropriately. We talk so much in Agile about ‘releasing customer value sooner’ yet, a great deal of the programmers I work with can’t really tell you what customer value is. For fun, ask around your work place and see what answers you get. Some people think it’s faster performance. Some think it’s newer technology. Customer value is that for which the customer is willing to pay the next dollar.

    With a mandate like that, a lot of the impending ‘functionality’ everyone is trying to quickly add to the system can be easily prioritized and in many cases an organization will find a large portion of the list that was generated from wherever, simply does not have a customer waiting with an open check book at the end of it. This is a simple rule of thumb and it will help eliminate much of the ambiguous ‘needs’ that seem to get added to development backlogs. Eliminating non-customer-value items is a huge benefit in these types of situations because it allows the programmers to focus on those items which are really important, and deliver real business value sooner.

    Reply
  4. Mike Cottmeyer

    Sachin… I think you will be hard pressed to convince the board, or the shareholders, to throttle back sales. That will get you fired. Got any other suggestions?

    ;-)

    Reply
  5. Mike Cottmeyer

    Peter… I think the first step will be highly dependent on your environment. For now, I would probably start on the BA/PO side of things with the clients I am thinking about. Next step might be short term QA help while we focus on reducing technical debt and automation on the Dev site. You might approach sales and ask them to back off committing to new features until the system stabilizes.

    Reply
  6. Mike Cottmeyer

    Mike… agree with you in principle, but often those sales come with contractual commitments. I think the solution has to go beyond simple prioritization. I use story mapping and the idea of an MMF. We have to aggressively decompose the work so we can sort out the smallest possible feature set that development can build to deliver the business value. Then deliver in priority order. To your point, i don’t want the dev organization driving that decision making (in most contexts at least).

    Reply
  7. Sudheer

    How about applying WIP limits across the board (sales, prod. mgmt, dev and qa) if you are hard pressed to recruit or train to start with

    And then Divert the forces to each area to resolve issues in parallel if something is blocked outside of WIP limits

    Trying to think lean :)

    Reply
  8. Matt Block

    Great post Mike! This describes the situation I was facing almost to a T and you are definitely correct about it being a “non-trivial” problem. There are many ways it can be approached all with their own pros and cons. I think the most important thing is that you just need to recognize it and start attacking the problem. I wrote a paper on our agile adoption process which, while it doesn’t really mention this problem directly, really walks the reader through the steps we took to address these issues. While I don’t think we are completely out of the woods, we are definitely in a much better position than before. The paper (Evolving to Agile) can be downloaded from the Resources section of my website (http://www.developmentblock.com/resources/) or anyone attending Agile2011 can attend the session at which I’ll be presenting it!

    Reply
  9. Balance the System First | Development Block

    […] By Matt Block, on August 3rd, 2011 Mike Cottemeyer had a great post over on LeadingAgile called Balance the System First.  In it he describes a situation in which many software companies end up. Over the years, sales […]

    Reply
  10. Balance the System First | Development Block

    […] Cottemeyer had a great post over on LeadingAgile called Balance the System First. In it he describes a situation in which many software companies end up. Over the years, sales have […]

    Reply