Alistair Cockburn, Humans and Technology
Humans and Technology Technical Report HaT TR 2006.04, Sept 6, 2006.
Engineering – software engineering in our case – is remarkably like manufacturing, once we shift to view the unvalidated decision as the unit of internal inventory. The parallels with manufacturing allow us to borrow from results in manufacturing theory to understand how to better structure our software processes, and the teams involved.
It is generally considered frivolous to compare software engineering with manufacturing. Manufacturing consists (so the argument goes) of making the same thing over and over, while engineering is about making something different each time. In software engineering, the design is the hard part, production is the easy part, often as easy as pressing CDs or publishing onto the internet.
Once we shift to considering decisions as the items being handed from person to person, gathered into larger and larger assemblies, then suddenly, there is a very real parallel between design and manufacturing. This is useful to us because manufacturing has been heavily studied over the last hundred years, and we can learn from their lessons.
In what follows, I shall talk about software development, but it should be clear that the same argument applies to every team design activity, including theatre, publishing, engineering, and much of business.
Waiting for Decisions
We start by recognizing that in team design activities, people wait on each other for decisions.
Figure 1 shows a simplified view of the dependencies between people in software development – it is missing the feedback loops, in particular (which are shown in Figure 2).
Figure 1: People wait on other people for decisions
In Figure 1, the dependency of one person on another is shown with a large gray arrow. The person at the head of the arrow is waiting for or being handed decisions from the person at the tail of the arrow. A small pyramid represents the actual decision being passed from one person to another.
In Figure 1, we see
- business analysts and user interface (UI) designers waiting for users and sponsors to decide what functions and design styles they want,
- programmers waiting for business analysts to work out the business rules, and UI designers to allocate behavior to different pieces of the user interface,
- testers waiting for programmers to finish their coding.
A nice thing about considering individual decisions as connecting people is that we can move away from stereotypes about how a company’s process or decision-making activities ought to look like, and instead focus on what it actually looks like – what decisions actually get made by which people, who is really waiting for whom.
There is no ideal software process any more than there is any ideal manufacturing process. Each company has its own strong-minded people, who make a disproportionate number of decisions that might, in other companies, be made by people in other roles. Each company has its own shortage of UI designers, programmers, testers, or even sponsors, which causes its process to have a certain characteristic shape – people working overtime, or perhaps sitting idle because other people can’t get their work done fast enough. Each company has its own reasons to have a large, external test department, or perhaps no test department at all.
Figure 2 shows the feedback loops in a common sort of organization. The feedback loops complicate the math, but don’t change the basic results we’ll be discussing.
:Figure 2: The decision-dependency network showing feedback loops
Different Bottlenecks, Different Processes
In any organization, we can find a backlog of decisions stacking up at some particular work group. The presence of that stack indicates a bottleneck, a place limiting the speed of the overall team.
Bottlenecks are of great concern in manufacturing, and have received much study. The obvious thing to do is to increase the capacity of the bottleneck group – hire more people, or better people, or get better tools, and so on.
Sooner or later, however, the organization hits its limit as to what it can do to improve the speed of the bottleneck group. At that point, what comes into play is the process definition itself.
Figure 3 shows three different, but fairly typical organizations.
- In the first (top) organization, there are not enough user interface and database designers to keep up with the work. We see decisions stacked up at their work centers.
- In the second (middle) organization, there are not enough experienced programmers, and work requests stack up in front of them.
- In the third (bottom) organization, the users and sponsors are notably missing from the discussion. What happens in these organizations is that the business analysts and UI designers end up making the business decisions, and then sending those decisions (or running products) back to the users and sponsors for comment. The picture shows those requests for review stacking up in front of the users and sponsors.
- The third (bottom) picture also shows the programmers and database designers sending decisions back and forth to each other. Both groups need to come to agreement on the domain model and how that will be represented in the code and in the database.
Figure 3: The optimal process varies with the decision-dependency network
There are two points to draw from these pictures:
- First, they should be using different processes;
- Second, these drawings help us to see how those processes should be different.
The organization at the top, assuming that they can’t or won’t hire more UI and database designers, should look at ways to have programmers and business analysts pick up sections of the UI designer’s and database designer’s work. Even assuming that UI design work is specialized, parts of that work can be automated, carried out by assistants, or handled by programmers.
In the second organization, the reverse is more the case. The programmers, being few and probably inexperienced, might need to have much of the problem “digested” for them as much as possible.
In a case that I participated in, we recommended that the business analysts write quite detailed use cases (not containing the user interface, but containing the business rules more explicitly than we otherwise might), and also the layouts of the data needs, plus discussions of different business scenarios. The business analysts then sat with their respective programmers as they started on each use case, and discussed the use case, the scenarios, and the data. They left the paperwork with the programmers, and made themselves available for discussions and tutorials as needed.
The process in this second organization was aimed at minimizing the trouble the programmers had to undergo to understand the problem at hand. This is quite different from the process in the first organization.
In the third organization, the process might call for prototypes and early samples to be produced and put in front of the users and sponsors. Since those people have the least availability, the material should be as fully prepared as possible. Also, since close collaboration between the programmers and database designers is required, those teams should be seated together, or at least have frequent meetings and joint design reviews.
The purpose of this article is not to make these recommendations, which any good manager or consultant would be likely to spot, but to show how they come from mapping and considering the bottlenecks in the flow of decisions through the organization.
Lessons from Manufacturing
To complete the mapping to manufacturing, we use the unvalidated decision as our unit of internal inventory, or “work in progress” (WIP). Each decision that has been made but has not yet been validated by someone (the sponsors, the testers, the marketplace) is still “pending”, still waiting to show its quality.
Move Inventory Out
The first lesson that we draw from experiences in manufacturing is to keep work-in-progress (WIP) down. This is as important in software development as it is in manufacturing, since every decision that is pending costs the organization money.
The value of decisions decays over time:
- Each “requirement” is a decision based on a business climate; when the business climate changes, the decision may become incorrect. If the software isn’t yet earning value for the company, the requirement is waste.
- An “architecture” is a decision based on technology and business. If the technology changes before the software is earning value for the company, those decisions are waste.
- Each line of code is a decision based on requirements, domain, technology and aesthetics. If anything causes it to become obsolete before the software is earning value for the company, it is waste.
To the extent that it is not earning value in the business, each decision loses value and quality with time. The more decisions stuck inside the pipeline, the more decaying “inventory” the organization is carrying.
Inventory stacks up quickly, even exponentially. Assume for reference an organization that is so fast that when a new requirment arrives, they can implement and deploy it by the next morning:
- The company with a 1-day turnaround has about 1 day’s worth of inventory lying around the office.
- The company with a one- to two-week turnaround has about 10 days worth of inventory lying around the office.
- The company with a quarterly delivery system (assuming they deploy from fresh requirements every quarter) has about 100 days of inventory lying around.
- The company delivering a 3-year project has 1,000 days of inventory (decaying) around them.
The message, in software as much as in manufacturing, is: get the inventory out the door and earning value! Find ways to shorten the pipe.
Move Small Amounts, Continuously
The second lesson to draw from manufacturing is to reduce the size of the transfer batch between working groups. That is, move small amounts, continuously.
Figure 4 shows two ways of transfering work from the programmers to the testers.
In the first case, the programmers hand over 100 lines of code (each week, let’s suppose). The testers get a regular weekly arrival rate of about 100 lines of code, and have to integrate and test them against the rest of the system and against the known defect log.
Figure 4: Feed systems run more smoothly with small transfer sizes
The actual size handed over will vary, of course, and the actual length of time needed to work through the new code will also vary. That variance is very important.
The lower part of Figure 4 shows the programmers handing about 1,000 lines of code to the testers (each quarter, that would be, to keep the rate of production about the same as in the upper picture).
The problem with the lower picture is that all 1,000 lines of code show up at one time. The responsiveness of the testing group suddenly becomes much more variable, with the large arrival of an unknown number of bugs of varying sizes.
Equally bad, when they start handing over bug reports back to the programmers, the programmers suddenly see a large spike on requests on their input queue coming from the testers (see the arrow in Figure 2, from the testers back to the programmers). The programmers are now juggling two input queues, requests for new features and requests for bug fixes.
Manufacturing lines have experienced and studied all the above, and concluded that these sorts of feed systems run best when small amounts of work get handed from one group to the next. The ultimate goal is to hand over just one part from one group or person to the next.
Toyota pioneered this idea in its lean or just-in-time manufacturing lines. They aim for continuous flow, the flow of just one piece of material from one person to another (what to do when a queue backs up is the subject of another lesson we’ll see).
It is not clear exactly what continuous flow might mean in software development. Some decisions affect large parts of the system, and some decisions can’t be validated for a long time. However, the experiences in manufacturing are backed up by both mathematical models and experiences in agile software development.
It is rare to find development teams able to deploy fresh requirements every week, but I have been able to find a few teams who both deploy weekly and have a low enough defect rate that they get only one or a few requests a day. On one team I talked with, a person was assigned each day, on a rotation, to handle any incoming requests, whether bug reports or requests for small enhancements. That person would stop other work, do the work and redeploy the system before rejoining the main group. The average time to redeployment was half a day. With such a small, steady flow of requests on the feedback queue, the team was able keep from being diverted from their main assignment.
The lean and agile manufacturing literature contains the recommendation to cross-train people at adjacent stations. The idea is that when a small bubble of inventory grows at someone’s input, the neighboring person having a spare moment steps over and works it down. In this manner, small variances in work flow can be evened out and not disturb the organization’s overall flow.
We see this in software development, when programmers help testers, user interface designers and business analysts, or when business analysts help testers. Unfortunately, programming is a technical enough activity that UI designers, testers and business analysts are unlikely to be able to help the programmers. Programmers can help other programmers, though. We see front-end developers, middleware developers and back-end developers helping each other in some companies when one of the groups has a sudden bump in work.
Extend the Network
All of the above ideas are good – so good, in fact, that a company using them soon finds that their bottleneck lies somewhere in their supply chain, whether their sponsors, their subcontractors, or their distributors. These companies start to draw the queuing or decision-dependency network for the larger system in which they sit, and start including their supply chain partners in their discussions.
Toyota is well known for working with its suppliers. Less well known are cases of software development groups doing it. The same team I referred to earlier, having the the daily programmer rotation for fixes and enhancements, also wrote automated acceptance tests for their subcontractor’s parts of the system. They reasoned that their time was better spent writing automated acceptance tests and catching bugs on arrival than debugging and finding those same faults in the integrated system when their supplier’s code broke it. The supplier was, of course, surprised but delighted to find they didn’t have to write the automated acceptance tests.
The lesson from Toyota and the other companies who are streamlining the wider network is: the wider the network of ‘we’, the faster we all go..
Who’s Writing About This?
Once you see the mapping between manufacturing and team design activities, suddenly a lot of literature becomes available.
Toyota’s production system, also called The Toyota Way, Toyota Production System, and lean manufacturing, is widely documented. The Toyota Way Fieldbook  is a good place to start.
The application of lean manufacturing principles to design work is described in Managing the Design Factory . The Poppendiecks have appled lean manufacturing principles to software development [3, 4]. Agile Software Development  contains an experience report from a software product company (Tomax) that is including its customers directly in it dependency network.
Goldratt wrote about bottleneck stations in manufacturing  and then widened the disussion to constraints in general (theory of constraints) . David Anderson applied the theory of constraints and queue size to software projects . I have written about strategies for dealing with bottlenecks that have reached their capacities .
It is not immmediately obvious that software development teams can learn from manufacturing. However, once we map the network of dependencies between people in a software development organization and make the shift to think of unvalidated decisions as comprising the “inventory”, then the parallels become startlingly clear.
We learn 6 lessons from the parallels:
- Drawing the decision-dependency network helps us see the bottleneck stations, where decisions-to-be-made are piling up.
- From the different decision-dependency networks in various organizations, and their varying bottlenecks, we can see how the optimal process varies from organization to organization.
- Move inventory out: Decisions decay over time, so it is important to find ways to shorten the pipeline from arrival of a request or decision to the deploymnet of the system.
- Move small amounts, continuously: Transferring large batches of inventory (decisions in our case) between workers causes unpredictable variations in the organization’s output. It is better to move small numbers of decisions more often. This reinforces the idea of incremental development, with the smallest increment size possible.
- Cross-train people: When people can help each other across specialties, they can move quickly to eliminate small bubbles in each others’ input queue, thus smoothing the organization’s output.
- Extend the network: By widening the network included in the dependency analysis and queue-size reduction, a company can smooth its own input stream and simplify its working.
- Liker, J., Meier, D. The Toyota Way Fieldbook, McGraw-Hill, 2005.
- Reinertsen. D., Managing the Design Factory, Free Press, 1997.
- Poppendieck, M., Poppendieck, T., Implementing Lean Software Development: From Concept to Cash, Addison-Wesley, 2006.
- Cockburn, A., Agile Software Development: The Cooperative Game, 2nd Edition, Addison-Wesley, 2006.
- Goldratt, E., The Goal,
- Goldratt, E., What is This Thing Called the Theory of Constraints?,
- Anderson, D., “Managing Lean Software Development with Cumulative Flow Diagrams,” May 2004, BorCon 2004 (online at <).” target=”_blank”>http://bdn1.borland.com/borcon2004/article/paper/0,1963,32096,00.html>).
- Cockburn, A., “Two Case Studies Motivating Efficiency as a "Spendable" Quantity”, Humans and Technology Technical Report HaT TR 2005.00..