Better Software Development Without the Hype

New workshop: Product Sashimi, Philadelphia, October 2

me@jbrains.ca (J. B. Rainsberger) — Sat, 04 Sep 2010 08:00:00 GMT

You have an idea for a product, but you’re not sure how to explain it to programmers, or you’ve tried, and they just don’t get it. We can help you.

Your marketing or sales department has sold vaporware to their customers, and you have to figure out how to deliver software that will delight them. We can help you.

You’re responsible for a project that was late before it started, and you have to ship something of value or the hammer’s going to come down. We can help you!

We can help you turn a nebulous product idea into a plan with some chance of success; or we can help you ship the most for the least from your already-way-too-big scope of work. Either way, it boils down to this: you will learn how to slice a product thinly.

We don’t know about you, but the very idea makes us hungry. For sashimi. That’s why we named this workshop the way we did. In a single day, we present to you a three-course menu of sashimi-inspired techniques.

First course. Essential feature areas: which major areas of the system are essential to delivering a complete, coherent product?

Second course. Concrete features: which specific features are most likely to get people to buy and use your product?

Third course. Executable examples: how can you describe each feature in a way that best helps programmers understand what to build?

Join J. B. Rainsberger and Bonnie Aumann and learn three key techniques to help you deliver high value for low cost and turn your product vision into reality soon.

We are offering this beta version of our workshop once, perhaps twice, at a drastically reduced price. Help us test and improve the workshop and get the best value possible for your time and money.

September 04, 2010 08:00

Squeeze More Business Value From Your Agile Transition

me@jbrains.ca (J. B. Rainsberger) — Thu, 02 Sep 2010 08:00:00 GMT

September 02, 2010 08:00

"Agile Design: Beyond the Basics" coming to Agile Eastern Europe in Kyiv, Ukraine

me@jbrains.ca (J. B. Rainsberger) — Tue, 31 Aug 2010 08:00:00 GMT

I would like to thank the organisers of Agile Eastern Europe to inviting me back to speak at their conference, and even more for inviting me to run my new master class, Agile Design: Beyond the Basics.

Bring a laptop, a problem, or even a code base, if you can! We will explore together how to make good design decisions in increasingly complex situations using the same simple design principles that work everywhere. Learn to solve complex problems with simple solutions.

I hope you’ll join me in Kyiv, Ukraine on October 6-7.

August 31, 2010 08:00 agile, design

Congratulations to the Pask Award 2010 recipients

me@jbrains.ca (J. B. Rainsberger) — Tue, 17 Aug 2010 08:00:00 GMT

Elisabeth Hendrickson

Liz Keogh

We'll write more about the recipients in the coming weeks.

August 17, 2010 08:00 agile, agile 2010

Where were the programmers at Agile 2010?

me@jbrains.ca (J. B. Rainsberger) — Sun, 15 Aug 2010 08:00:00 GMT

I just came back from Agile 2010, which I rather enjoyed. We had some intense discussions, and plenty of whining sessions, about the relative lack of content for programmers. I found out that about 1/6 of the conference attendees identified themselves as “developers” and about 1/5 identified themselves as “consultants”. If we suppose that even half the consultants are programmers, then only about 1/5 to 1/4 of the attendees had medium-to-strong interest in deep content for programmers. It turns out that the number of such sessions amounted to about 1/5 of the program, and by all reports, most had lower-than-expected attendance. If the conference offered enough content for programmers, then I have to ask two questions:

Why didn’t more programmers attend Agile 2010?
Why didn’t more of the programmers who attended Agile 2010 go to the deep technical sessions?

If you’re a programmer, then why did you choose not to attend Agile 2010?

If you’re a programmer who attended Agile 2010, then why did you choose not to attend many/any sessions meant for programmers?

If you manage programmers, or otherwise approve the budget for them to attend Agile 2010, then why did you decide not to send them/let them attend?

I appreciate any feedback you’d be willing to provide. I intend to use this information to decide how I will spend my energy proposing technical sessions for future conferences.

August 15, 2010 08:00 agile, agile 2010

Design your own TDD learning experience

me@jbrains.ca (J. B. Rainsberger) — Thu, 05 Aug 2010 08:00:00 GMT

Join me in Chicago in September to learn whether test-driven development will work for you. In this course, you will learn the secrets of modular design from one of test-driven development’s master practitioners. Bring your laptop and be prepared to change the way you write software.

August 05, 2010 08:00 junit, refactoring, design, extreme programming, TDD From the Beginning

Before APLN, before agile kanban, before "The Apprentice," there was...

me@jbrains.ca (J. B. Rainsberger) — Fri, 30 Jul 2010 08:00:00 GMT

Can you?

Can you produce clean code that works, while the pressure is on and the world is watching?
Can your stomach handle putting your programming skills on display for the entire conference to see?
Can you fly through your IDE at the speed of light?
Will you wither under the relentless taunting of top agile designers dissecting your every move?
Do you want your chance to be recognized for your prowess at programming, for your relentless refactoring, and for your tenacious TDD spirit?
Do you really write code you’re proud to put on the fridge? How about on the big screen in front of everyone?
Do you think you can survive three minutes in the hot seat?

Can you handle … the Extreme Fishbowl?

Eight years after the original event, the fishbowl is back—bigger and better than ever. The customer wants clean code that works, the developers want to exhibit their programming skills (while not getting fired), and the audience wants a good time while they eat.

Come see how much can get done in 40 minutes of intense, focused agile development.

What it is

Extreme Fishbowl celebrates the art of team-based, agile programming. The fishbowl holds four programmers, a customer, and a commentator, all working with real live code. Conference attendees can either watch or participate in this high-stakes development situation where functionality, quality, and process all matter. It’s a lot of fun!

Your customer wants clean code that works, he wants it now, and he knows what it looks like!

How it works

Each day during the lunch block, participants line up to await their turn in the fishbowl—a highly-visible, high-pressure development situation with everything on the line (including their reputations).

Wait for others to be “fired” until your turn rolls around. Then get seated quickly with your pair and await the ride of your life. You contribute code and tests to the ever-growing program, under the eagle eye of the demanding and occasionally capricious customer. Feedback is omnipresent, with all of the action shown on two big screeens and a running play-by-play commentary from the host.

As analysts dissect your every move, the audience watches in anticipation. One misstep—one too many red bars when you needed green—and it could be your last!

A demanding, capricious customer who knows more about good design than you do… it’s your worst nightmare!

Who will be there?

Your crotchety customer and relentless reviewer: J. B. Rainsberger
Your hilarious host and penetrating pundit: Jeff Nielsen
Surprise guest analysts, plucked from among the best agilistas in the business
You!

July 30, 2010 08:00 agile, agile 2010

Three Steps to a Useful Minimal Feature

me@jbrains.ca (J. B. Rainsberger) — Fri, 16 Jul 2010 08:00:00 GMT

Recently, in the mailing for Steve Freeman and Nat Pryce’s book Growing Object-Oriented Systems Guided by Tests, I followed a discussion that included this comment:

I guess the skill is knowing in what makes as small as possible but valid slice (as you say in the book)

Even though I’ve written before about splitting stories, I have refined my ideas about how to deliver a useful, significant, but small slice of software. I use a simple technique which I describe this way:

Write out any, and I mean any, meaningful end-to-end scenario in detail with concrete values at every step.
Now that you’ve chosen one real scenario, go to each step in that scenario and ask the question, “What would I need to assume to eliminate this step?” If you find those assumptions make for a reasonable scenario, then use that assumption to simplify the scenario.
Repeat step 2 until exhausted or unable to come up with a simplifying assumption with five minutes’ thought.

I’ve used the example of online bill payment in many of my classes and applied this algorithm. You’d be surprised how simple, but useful a bill payments system one can build.

In fact, let’s look at this example in a bit more detail.

What does a typical bill payments system look like? I imagine that TD Canada Trust has a fairly representative system, so I’ll use that as my example.

First, select the “Pay Bills” option.

Next, select the account to use to pay the bill.

Next, select the bills you’d like to pay. Once in a while, I want to pay multiple bills at once, such as when my business has to pay the property taxes on a handful of rental properties.

Next, enter the amount you want to pay, the date on which to pay it, the account from which to pay it (why again?) and whether you want to repeat this payment automatically.

Verify the payment details: the amount, the account, the creditor, the date, and the system schedules the payment.

From this, I can describe the scenario in a form that looks like an executable example:

Pay a bill online

Given that Joe has already logged in
Joe says "I want to pay a bill"
Joe selects chequing account 12345 from which to pay the bill
Joe selects Visa bill with account ending in 2222 as the bill to pay
Joe says "I want to pay $5,000"
Joe selects a date 14 days from now as the date on which to pay the bill
Joe selects "only once" as the frequency with which to pay the bill
After Joe sees a summary of the bill payment he has asked to schedule, he says "I confirm that I want to pay this bill"
Now the system schedules to pay the bill as requested and sends Joe an email to confirm the transaction with a link to cancel or change the scheduled payment

Apart from the numbers, this scenario perfectly accurately reflects how I pay my bills online, although I only wish TD Canada Trust would send me the confirmation email I threw in as the system’s response. We have completed step 1 of the algorithm: we have specified a complete scenario with concrete values at every step.

You'll notice that we didn't specify Joe's username and password. We don't intend to re-test logging in here, so we don't bother with those details. We will have tested that elsewhere.

Now we move to step 2 of the algorithm: looking for assumptions we could make about paying a bill online that would eliminate steps in the process. To do this, we have to be prepared to sacrifice any semblance of a decent user experience. Don’t worry: once the Walking Skeleton runs, you’ll be able to add all the bells and whistles that will make this feature a pleasure to use. For now, we want to eliminate any detail that distracts us from connecting our feature to the key interfaces it must deal with. In this example, I know I want to expose an HTTP interface to clients (eventually the web) and that I need to connect to the Big Ugly Banking System, but beyond that, I don’t know that anything else matters. Within this context, then, we can start making our simplifying assumptions.

That is, until someone remembers that, being a bank, we need to keep a strictly accurate record of all transactions. That means we should add a final step to the scenario: the system records the transaction in its log. While some might consider this a superfluous detail, banking regulators would call it quite essential, and so I find it hard to ignore. This means that we have a third essential interface to which to connect: the transaction logging facility. For our purposes, I’ll assume that we have one and that it has the usual properties: transaction posting date, description, and debit or credit amount, who performed the transaction and when.

This itself makes me ask whether we need to log the transaction yet, because in our scenario we’ve scheduled a payment, and not made one. Scheduling a transaction leads to issues of canceling, editing, and building a process that completes the transaction on the day the customer scheduled it. This leads to our first simplifying assumption: Joe pays the bill immediately.

This simplifies the scenario because we no longer need Joe to tell the system when to pay the bill: the system always pays the bill immediately. Our revised scenario looks like this:

Pay a bill online

Given that Joe has already logged in
Joe says "I want to pay a bill"
Joe selects chequing account 12345 from which to pay the bill
Joe selects Visa bill with account ending in 2222 as the bill to pay
Joe says "I want to pay $5,000"
Joe selects "only once" as the frequency with which to pay the bill
After Joe sees a summary of the bill payment he has asked to schedule, he says "I confirm that I want to pay this bill"
Now the system schedules pays the bill as requested and sends Joe and email to let him know that the bill was paid

I like to start at the top and look for simplifying assumptions. First, I see that Joe has to select the account from which to pay the bill, which implies that the system presents a list of accounts to Joe, which we recognize we need to do, but not in the Walking Skeleton. Ultimately, Joe simply needs to specify the account number to debit to pay the bill, so for now we’ll make him type that in. Our revised scenario looks like this:

Pay a bill online

Given that Joe has already logged in
Given that Joe has already logged in
Joe says "I want to pay a bill"
Joe says "I want to pay from account 12345"
Joe selects Visa bill with account ending in 2222 as the bill to pay
Joe says "I want to pay $5,000"
Joe selects "only once" as the frequency with which to pay the bill
After Joe sees a summary of the bill payment he has asked to schedule, he says "I confirm that I want to pay this bill"
Now the system schedules pays the bill as requested and sends Joe and email to let him know that the bill was paid

Next, I see that Joe again has to select the bill to pay, which implies that the system presents a list of bills to pay. We could simplify this by requiring Joe to enter the payee identification number and the account number, even though this means saddling Joe with knowledge of payee identification numbers. In particular, the system neither has to store nor present a list of bill payees, and Joe doesn’t need to register a creditor before paying them.

I'm making up this notion of payee identification numbers because I don't know how banks really implement this. I imagine whatever they do, it boils down to companies registering as payees for bill payments, which results in issuing them some kind of identification number. If some kind soul wants to educate me on how this really works, I'd gladly edit the article to bring it closer to the banking industry's real implementation.

Our revised scenario looks like this:

Pay a bill online

Given that Joe has already logged in
Given that Joe has already logged in
Joe says "I want to pay a bill"
Joe says "I want to pay from account 12345"
Joe says "I want to pay to payee number 66666"
Joe says "I want to pay to account number 2222"
Joe says "I want to pay $5,000"
Joe selects "only once" as the frequency with which to pay the bill
After Joe sees a summary of the bill payment he has asked to schedule, he says "I confirm that I want to pay this bill"
Now the system schedules pays the bill as requested and sends Joe and email to let him know that the bill was paid

Next, I notice that Joe has to specify the amount to pay, and I can’t think of how to eliminate that detail without complicating matters. It does make me think about potential future features, such as “pay balance off in full” and “pay minimum payment required”, which I note down before returning to this scenario. Joe will simply have to tell us exactly how much to pay towards the bill.

Next, I notice that Joe has to confirm that he only wants a one-time payment. We can eliminate this detail by assuming that Joe can only pay the bill once. We know that customers want recurring payments, but that only distracts us from implementing the Walking Skeleton. We can eliminate this step, and our revised scenario looks like this:

Pay a bill online

Given that Joe has already logged in
Joe says "I want to pay a bill"
Joe says "I want to pay from account 12345"
Joe says "I want to pay to payee number 66666"
Joe says "I want to pay to account number 2222"
Joe says "I want to pay $5,000"
After Joe sees a summary of the bill payment he has asked to schedule, he says "I confirm that I want to pay this bill"
Now the system schedules pays the bill as requested and sends Joe and email to let him know that the bill was paid

Next, I notice that Joe has to confirm the bill payment before the system will process the payment. While this step might seem essential for security reasons, remember that we don’t necessarily have a graphical web interface for our Walking Skeleton implementation, and so we might not even have the opportunity to ask for confirmation of the bill payment. On this basis, we eliminate this step by assuming that Joe has looked over the details before pressing the button to pay the bill. Our revised scenario looks like this:

Pay a bill online

Given that Joe has already logged in
Joe says "I want to pay a bill"
Joe says "I want to pay from account 12345"
Joe says "I want to pay to payee number 66666"
Joe says "I want to pay to account number 2222"
Joe says "I want to pay $5,000"
Now the system schedules pays the bill as requested and sends Joe and email to let him know that the bill was paid

I can’t see any further simplifications, so I choose to stop here. I suspect this constitutes a close-to-minimal protocol for the “pay a bill online” feature. The programmer in me sees this as a single message, which pleases me, because of the simplicity of the interaction. The customer in me can see clearly all the extra stories we need to complete to make this feature available for public use, which makes planning easier. It feels like a win for everyone, except perhaps for Joe, who has a crappy interface to work with.

Now that we have a Walking Skeleton bill payment feature, we can identify the stories we want to deliver beyond the simplest case, and can decide which ones we need to roll this feature out to paying customers.

Let Joe choose from a list of available payees which company to pay
Remember the payees that Joe has previously paid and present them as “favorites” so he doesn’t have to search for them
Remember the payee accounts that Joe has previously paid so that he doesn’t have to enter them each time
Let Joe delete accounts he no longer needs to pay
Give Joe the option of paying the minimum payment required by the payee
Give Joe the option of paying the full balance owing
Let Joe schedule his payment in the future
Let Joe cancel pending payments
Let Joe change pending payments
Send a reminder to Joe to pay a bill he has paid at least three of the past six months (try to detect a recurring payment)
Let Joe schedule a payment to recur every month (same day each month)
Notify Joe in advance of automatically detected recurring payments and ask him if he wants us to pay the bill for him
When emailing Joe about a bill payment, include links to review the scheduled payment, change it, or cancel it, if appropriate

I imagine we could come up with more together, but I find one common thread with all these stories: once we implement the Walking Skeletion, we can implement most of these stories independently of the others. We know that more independent stories means greater opportunities to change priorities as needed as well as greater opportunities to drop features in favor of other more lucrative options. Once again, everyone wins.

Would you like to work with J. B. Rainsberger to realize revenue sooner and lower costs from delivering software? Schedule a workshop with J. B. today.

July 16, 2010 08:00 stories, planning, design, article

Best of... June 15, 2010

me@jbrains.ca (J. B. Rainsberger) — Thu, 17 Jun 2010 08:00:00 GMT

The tweets I found most interesting or useful in the past week.

@mfeathers: You don’t get abstraction until you’ve squeezed every bit of duplication from a piece of code walked away and found some more.

“There are no simple tricks for eliminating blame. You simply must stop doing it.” –Juran Institute, Inc.

Back to Basics: Efficiency v. Effectiveness by Matt Heusser

@ByronKatie: Stress is an alarm clock that lets you know you’re attached to something not true for you.

June 17, 2010 08:00

Some examples of programming in pairs

me@jbrains.ca (J. B. Rainsberger) — Tue, 15 Jun 2010 08:00:00 GMT

I had the pleasure of pairing with Naresh Jain during Agile India 2010 in Bengaluru and Mumbai in January 2010. Naresh recently uploaded some video that I’d like to share with you.

This is how we sometimes end up pairing:

This is how we like to think we pair:

June 15, 2010 08:00 agile, people, communicating effectively

Video: Education and Software Craftsmanship

me@jbrains.ca (J. B. Rainsberger) — Sat, 03 Apr 2010 08:00:00 GMT

In February 2010, MosaicWorks of Bucuresti, Romania, hosted Corey Haines and me to teach the world’s best introduction to Test-Driven Development. After the class ended, they interviewed us on the topics of software development education and the craftsmanship movement. Please enjoy.

Part 1:

Part 2:

Part 3:

April 03, 2010 08:00

Making decisions doesn't have to be so hard

me@jbrains.ca (J. B. Rainsberger) — Wed, 10 Mar 2010 08:00:00 GMT

I enjoy collaborating on decisions, but only with groups that agree to use a technique I refer to as consent-based decision making. I might not use that term exactly as the coiners intended, so let me explain what I mean. I characterize consent-based decision making by putting forth a proposal, then looking for reasoned objections. When no-one raises such an objection, we accept the proposal. I find this style of decision making quicker, easier, and better for team unity than typical approaches.

Consent-based decision making contrasts sharply with a typical decision-making exercise, which tends to follow these steps:

Present a need or problem.
Present options.
Solicit more options from the group.
Discuss the merits of each option in detail.
Propose solutions.
Combine the proposed solutions into a kind of hybrid solution to which everyone can assent.
Ask for any last-minute objections.
Decide.

I feel tired trying to make decisions this way, and it seems that the fatigue rises with at least the square of the number of people involved. Not only do I find it difficult to make decisions this way, but that difficulty encourages me to exclude people who might otherwise have valuable input. It puts me in a place of wanting to prune ideas, rather than generate them. You can understand why I’d want to avoid this kind of consensus building.

Some objections

When I have taught consent-based decision making to my clients, some of them have pointed out that it leads to a particularly negative culture: “Don’t bring me problems; bring me solutions.” I agree that, practised mindlessly, that could happen, but because I insist we practise mindfully, I think we can avoid this problem. Still others have pointed out that this style of decision-making stifles creativity because it intimidates people who want to point out a flaw in a proposal without necessarily having a better proposal. Again I agree, but we can mitigate this risk by looking at one of consent-based decision making’s greatest strengths: separating generating ideas from selecting a solution.

I remember dozens of meetings in which I participated in making decisions by building consensus, specifically how inconsistently engaged I felt. I would enter some of these meetings with a desire to generate ideas, brainstorm, and explore solutions; and I would entry other of these meetings tired of sifting through ideas, craving to decide on a course of action. I can’t tell you why I felt the way I felt, but rather just that it varied, and that I felt it strongly. Sometimes I wanted to expand the solution space, and other times I wanted desperately to contract it. So far, I don’t see a problem with that, but there are, of course, other people.

When you and I enter a decision-making meeting with different goals, we create problems for each other. When you want to generate ideas and I want to select a solution, we fight for air time, for space, and indeed for life… at least, it feels that way to me. I struggle to bring us to a sensible solution, and as my wish comes close to coming true, you trample on it with yet another pie-in-the-sky idea. Worse, your idea might fit perfectly, but I simply won’t see it that way. I will interpret your every new idea as an attempt to prolong the agony, whereas you will interpret my every attempt to choose a solution as an attempt to shut you down. Result? War. Root cause? Cross purposes. Remedy? Alignment. (Surprised?)

Two goals, two meetings

Consider, instead, having two separate meetings: one to generate proposals, and one to select a proposal. You might find that that helps.

In the first meeting, we generate ideas, brainstorm, solicit opinions, run impact studies… whatever we need to do to generate proposals. The people who come to this meeting might or might not care about making the decision. Whatever happens, we can feel certain that whoever shows up wants to lend their ideas to the group, and most importantly, we ignore any attempt at choosing a solution. We agree in advance to ignore them, because we have a different goal in this meeting. We agree not to chastise people for attempting to choose a solution because they form part of our natural impulse to jump to a conclusion. We agree to recognize each other’s humanity by allowing each other to compare or rank solutions, but we generally ignore those attempts in a quiet, friendly manner. (See Ask Why, But Don’t Answer for an explanation.)

The second meeting starts with a proposal. The group may ask clarifying questions, but by now we shouldn’t need to ask too many. The Proposer than asks the group to vote, which the group does by signaling thumb up, sideways, or down. A thumb up means “I accept the proposal”. A thumb sideways means “I will go with the rest of the group”. I thumb down means “I reject the proposal”. Of course, if you reject the proposal, then you must make a counter-proposal right away, otherwise the group feels free to ignore your vote. The group repeats this process until either it makes a decision or reaches a deadlock. If it reaches a deadlock, then we immediately adjourn the meeting and schedule another one to explore the competing proposals in depth. Why schedule another meeting? In part, to discourage people from rejecting a proposal just for the sake of rejecting it, and to give those people an opportunity to sleep on it before beginning another round of brainstorming.

No silver bullet

Naturally, people could abuse this system. And yes, when we have to make particular tough decisions, this system could take longer than the consensus-building approach. Even so, for more routine decisions, consent-based decision making works more quickly and easily, and I’d rather make easy things easy and hard things possible than optimize for the hard decisions.

March 10, 2010 08:00 people, article, coaching

Ask why, but never answer

me@jbrains.ca (J. B. Rainsberger) — Sun, 07 Mar 2010 01:12:30 GMT

Sometimes I go on project archeological digs with my clients through their projects, often focusing on their code bases. We sometimes run a “code bazaar”, where we set up a half-dozen machines with code to dig through, then walk in pairs from machine to machine, digging and taking notes. During these digs, I institute a key rule:

Ask “why”, but never answer.

What kind of “why” question? Why, the most perplexing one: Why the $%@# did we do this?! We feel free to ask the question, but we don’t answer it. If someone feels the impulse to answer it, someone says, “We don’t go there.”

This rule serves a dual purpose. By never answering “why”, we avoid the exercise devolving into wallowing and blaming. We avoid counter-productive actions like justifying, going on the defensive, and shifting the blame. This allows us to have an open, frank discussion of the project in a safe environment. Why, then, do we permit people to ask “why”? Simply put, we don’t deny human impulse.

You’ve experienced it. You’ve reviewed code, screamed “What the fuck?!” and shook your head. I recently read code that made me angry enough that I had to stand and walk around to calm down. (It assigned a field to itself. Don’t ask.) I understand the power of the impulse to recoil, then demand of the universe how it could allow such a thing to happen. For this reason, we permit each other to ask “why”, and even to do so in a loud, obnoxious, pained tone of voice. We simply don’t answer that question, and if someone tries, we remind them that “we don’t go there.”

Try it the next time you have to rummage through code, documents, or some other project artifacts as a group. Ask the group to agree on one rule: you can ask “why”, but you can’t answer it.

March 07, 2010 01:12

You're probably sabotaging your people's training

me@jbrains.ca (J. B. Rainsberger) — Thu, 04 Mar 2010 08:00:00 GMT

You want to make a good investment in your people, and you know they need training, but when you invest thousands of dollars in courses without giving your people the slack they need to apply what they learn, you’re just being cruel.

Now granted, you don’t mean to act cruelly, but you have to take responsibility for the fact that when you give someone a chance to learn, but not to practise, you not only waste their time, but you tear down morale. Some of your people will even interpret your move as shallow, transparent appeasement. I’ve seen it, and I’ve felt it.

Let me explain.

First, let me start with a diagram that you absolutely need to know. I didn’t invent it, and I have definitely simplified it, but I believe I’ve kept its essence intact. This diagram graphs productivity against time as we learn a new skill.

When you provide your people training, you start them off at the beginning of this graph, with whatever productivity level they have in that area. If you wanted to train your people in design techniques, or a new programming language, or a new programming platform, then you could interpret the productivity level in terms of features delivered per week. If you wanted to train your people in negotiation techniques and interpersonal communication, then you could interpret the productivity level in terms of complaints about others, fights, or the general quality of discourse you perceive in meetings and during daily work. However you choose to interpret this graph, it works out the same for just about any measure of productivity. Before training, your people find themselves at the far left, at some baseline level of productivity.

After training, your people start to incorporate what they learned into their work. As they practise, they learn to execute the new techniques correctly, with uneven results. Once they become comfortable with the basics, they exhibit some improvement, but over weeks or months, they reach the first plateau. At this point, they have learned most of what they will learn from the first set of ideas and techniques they try.

Next, they incorporate some more of what they learned into their work. This time, their productivity decreases for a while as the new techniques they try clash with what they already know. During this time, they don’t know which techniques work best in which situations, so sometimes they choose well, and other times they choose poorly. They generally struggle integrating the new ideas and techniques into their work. Eventually they reach a point where the new techniques begin to mesh well with what they know, they see new benefits that they didn’t see before, their confidence improves, and their productivity resumes its increase, past their previous highest level, on to higher levels than they’d experienced before. This continues until they reach the next plateau, then the cycle begins again: struggle, bottom out, improve, plateau. This cycle continues indefinitely… unless you get in their way.

Saboteur? You?

I’ve seen two major categories of errors that managers make when they encourage their people to develop new skills: giving them no slack to learn, and panicking when they bottom out the first time. You might not realize you do these things, so watch for them, because I routinely see otherwise thoughtful, intelligent managers ruin potentially game-changing improvement programs by doing one of these two things. Let me clarify what I mean.

I’m going to tell you a story

I remember when I first tried to use the Getting Things Done system to manage my own work. I spent dozens of hours creating projects, contexts, and next actions. I think my first task review took an entire day, as I questioned at every step how to review tasks “correctly”. I also remember spending plenty of time learning to use first iGTD, then later OmniFocus, struggling with how to synchronize tasks with my calendar, and generally feeling my way around the system. I remember mechanically clipping parts of email into an OmniFocus task, creating a project for it, then responding to the email and marking the OmniFocus project as “completed”. I remember thinking that this couldn’t possibly improve my effectiveness, as it now takes a handful of steps simply to reply to an email, when it used to take only one. Still, I knew that unless I practised the steps, I’d never manage to execute them deftly, so I invested the time I needed to practise. For that, I needed to build some slack into my schedule.

Building slack into my schedule meant disappointing some people. It meant letting some revenue-realizing activities slip. I invoiced clients later, I paid bills slightly after they came due, and I turned down some opportunities to market myself that would surely have resulted in an increase in sales. I knew I had to do this, because if I didn’t improve how well I executed the work I needed to do, I would never clear the ever-lengthening backlog I had to complete. (Before you comment, I did consider throwing away the bottom 80% of my backlog, and could manage to throw away only about 20%. This helps you understand how far behind I fell in completing this work.) Even in a desperate situation, with wolves knocking at the door, I invested the time I needed to learn how to work more effectively, because if I hadn’t, then I would have continued struggling while even bigger, more ferocious wolves found me and knocked longer and louder.

Turning the first corner

Over several weeks, I noticed the first hint of what Getting Things Done promises: the feeling of having a trusted system that helps you ensure you do whatever you need to do. I started to see tasks show up a week after I’d added them, because they started to come due. Unfortunately, while I saw some early wins, I feel into a deeper trap: due dates. Knowing the urgency of my backlog items, I’d set tentative due dates for the vast majority of the tasks that I felt I needed to complete within two months. Naturally, since I had a blind spot for identifying all the tasks I needed to complete a project, that meant I had hundreds of tasks with due dates, averaging about fifteen per day. Every day when I looked at that list, I felt defeated, but I tried to soldier on. I fell into a rut of completing about five tasks, while deferring five more to “three days from now” (or worse, Friday) and, at the end of the day, pushing the last five to the next day. Within two weeks I went back to bed, feeling depressed about having over 40 “urgent” tasks to complete in a single day. It seemed hopeless.

At this point, I could have filed for temporal bankruptcy, abandoned Getting Things Done, thrown away the task list I’d spent dozens of hours crafting, and wallowed in my own inability to make progress on my backlog. I had some serious deadlines looming, which then passed, and I began to suffer so much from the stress of the wolves at my door that I reach my wit’s end. I could have simply quit, and for a while, I did.

Summoning the courage to try again

After a few weeks of fighting fire after fire, interspersed with hours spent in a mild vegetative state in front of the TV, I resolved to try again. I had read a great article about why high achievers procrastinate, and it resonated with me. At the same time, I ended up spending a week at my wife’s family cottage, re-reading Getting Things Done, then reading The Four Hour Work Week. Armed with more information, more ideas, and renewed enthusiasm, I cracked open OmniFocus, performed a thorough task review, threw away 90% of my due dates, and moved most of the projects into “Someday” and “Maybe” folders. This took over six hours spread over two days. After I got back to the office and resumed completing tasks, I noticed how much more I felt in control of my work. Within a month I had made more progress on my backlog than I had in at least two years. I felt better, but noticed a new problem: due dates started sneaking up on me.

Into the abyss once more

Since I had stopped setting due dates for most tasks, I found that some due dates began sneaking up on me. Some deadlines blew right past me. I felt a momentary and mild depressive state as I wondered whether I had chosen incorrectly in removing all the due dates. I read some more, thought some more, and performed another very thorough task review. After a week I decided to experiment with adding due dates only when I absolutely knew the date on which a task came due, then measure the results. I would occasionally agonize during a task review about whether I needed to put a due date on a certain task. Sometimes I decided in a few seconds, and sometimes it took me fifteen minutes, because I didn’t want to do this mindlessly. I needed to avoid falling back into the old habit of deferring tasks that had due dates, but didn’t really need doing by that date. I needed not to see 40 tasks due on a Friday ever again. I persisted and, since then, I gladly report that I get good results from my use of Getting Things Done: I still have the occasional hiccup with a due date sneaking up on me, but it happens once every couple of months. For the most part, I do what needs doing and aggressively look for ways to delegate what others could capably do for me. Services like Your Man in India and elance.com help me with that. I still have things to learn, but I’ve reached a pretty comfortable plateau where my productivity level appears to have matched my current workload.

And then…?

So why did I tell you that story? I hope to demonstrate the power of having slack to apply what you learn when you learn it, and the power of not panicking when things start to go wrong. Specifically, I have this advice for you:

If you plan to provide training to your people, you need also to provide about 20% slack time for them to practise what they learn.
If you cannot provide your people with 20% slack time, then do not schedule any training yet. Instead, figure out how to get them the slack time they need.
If you have currently scheduled training for your people and they won’t have the slack time they need to practise what they will learn, then balance the cost of canceling the training against the cost of getting them that slack time between now and when the training will begin.
After your people complete their training, work with them to build a charter related to the way they will use what they learn. This consists of a time-boxed experiment with a single measure of progress and a single goal related to that measure.
When you choose the length of time for your time-boxed experiment, make sure make it long enough to get past the first bottoming-out phase working towards the second plateau.
When your people show that they continue to struggle even half-way through the experiment, don’t panic! Trust the system, and maintain your commitment to invest in the entire experiment. Continue to measure progress, and make sure to check in with your people frequently and regularly, but whatever you do, don’t panic.

If you’d like to learn how to help your team adopt new practices effectively, schedule a private course with J. B. During your initial conversations with him, he will walk you through chartering how your people will use what he teaches them, help you figure out how to give them the slack time they need, and even give you tips on how not to panic.

March 04, 2010 08:00 people, training, coaching

"Integration Tests Are A Scam" Is A Scam

me@jbrains.ca (J. B. Rainsberger) — Sun, 14 Feb 2010 13:53:20 GMT

I made this announcement at a TDD course in Bucuresti this week, and I wanted to make it here: I now fully and freely admit that when I use the term “integration tests” I confuse people unnecessarily, and so I will stop.

As a result, you will notice me change from “integration tests” to “integrated tests”, because I believe the latter term better fits the meaning I intend to convey as well as avoids confusion with what everyone else means by “integration tests”. I agree to reserve the term “integration tests” for tests that focus on checking the integration points between subsystems, systems, or any nontrivial client/supplier relationship. Integration tests might be integrated tests, and might be collaboration tests. Your choice.

I apologize for the confusion I created, and appreciate you for hanging in there while I refactor my considerable library of legacy articles. That will take time and I can’t make it my full-time job.

February 14, 2010 13:53 integrated tests are a scam

What sessions do you want to see at Agile 2010?

me@jbrains.ca (J. B. Rainsberger) — Thu, 04 Feb 2010 13:56:41 GMT

I have already submitted one session to Agile 2010, and I don’t know what else to submit for the year. I’d like to leave one slot open for a potential collaboration (hint, hint), but as for a solo session, what do you think I could present or facilitate that you’d spend 90 minutes of your precious time attending? I’d prefer to do something new, which means no “Integration Tests Are A Scam” and no “XP: My Greatest Misses”, but that doesn’t mean that I couldn’t run a session on this topics.

You could help me by outlining what you might see in a session description in the Agile 2010 program that would attract you over the other sessions. Perhaps I can deliver that very session!

Thanks for your help.

February 04, 2010 13:56 speaking, agile 2010

Using integration tests mindfully: a case study

me@jbrains.ca (J. B. Rainsberger) — Sun, 31 Jan 2010 08:00:00 GMT

Gus Power commented about the way he uses integration tests in his work.

Interesting series of articles & comments. I also read Steve Freeman’s article in response to the same topic. It’s got me thinking about how we work and I thought I’d take the time to describe it here.

You define an integration test as “… any test whose result (pass or fail) depends on the correctness of the implementation of more than one piece of non-trivial behavior.” We have many such components that exhibit such non-trivial behaviour in the products we create, many of which are not developed by us. And we have integration tests to verify they work. I’m not just talking about 3rd party libraries and frameworks here, I’m referring to the whole system: caching layers. load balancers, DNS servers, CDNs, virtualization etc. When we build software it only becomes a product or service for our users when it has been deployed into a suitable environment; an environment that typically contains more than just the software we have written and packaged. Since our users’ experience and perception of quality result from their interaction with a deployed instance of the whole system, not just their interaction with the software at a unit level, we have come to value end-to-end integration testing. I believe there’s merit in testing these components in symphony and will attempt to clarify what kind of integration testing I’m talking about.

For a given piece of functionality we write an executable acceptance test in human readable form (for web projects we typically use some domain-specific extensions to selenium, for services we have used FIT and it’s ilk, sometimes we roll our own if there’s nothing expressive enough available). We run it against a deployed version of the application (usually local though not always) which typically has a running web/application server and database. The test fails. We determine what endpoint needs to be created/enhanced and then we switch context down into unit-test land. A typical scenario would involve enhancing a unit test for the url mappings, adding one for the controller, then one for any additional service, domain object etc. When we’re happy and have tested and designed each of the required units we jump back up a level and get our acceptance test to progress further. The customer steers the development effort as he sees vertical ‘slices’ of functionality emerge. The acceptance test is added to a suite for that functional area. The continuous build system will then execute that test against a fully deployed (but scaled down) replica of the production environment, with hardware load balancer, vlans, multiple nodes (session affinity) and so forth. Any additional environmental monitoring (e.g. nagios alerting) is also done as part of this development effort and is deployed into the test environment along with the updated code.

Setting up the infrastructure to do this kind of testing takes investment, both initial and ongoing. The continuous build needs to be highly ‘parallelized’ so you get feedback from a checkin in 10 mins or less (we’re heavy users of virtualization, usually VMWare or OpenVZ). The individual acceptance test suites need to be kept small enough to run quickly before check-in.

Benefits of this approach
* The continuous context-switch between acceptance test and unit test is key to our staying focused on delivering what the customer actually wants.
* The customer has multiple feedback points that he can learn from and use to steer the development effort.
* It confirms that the whole system works together – networking, DNS, load balancing, automated deployment, session handling, database replication etc.
* We create additional ‘non-functional’ acceptance tests that automatically exercise other aspects of the system such as fail-over and recovery.
* Upgrades to parts of the system (switches, load balancers, web caches, library versions, database server versions etc.) can be tested in a known and controlled way.

We’ve caught a number of integration-related issues using this approach (a few examples: broken database failover due to missing primary keys, captcha validation not working due to a web cache not behaving correctly, data not persisting because one database server had the wrong locale) and stopped them before they have reached our users. We have used the feedback as a basis for improving our products and their delivery at a system level.

OK this reply has now become far too long :-/ It would of course be good to discuss this in person sometime :)

—Gus Power

Thanks for the substantial comment, Gus. For those who don’t know Gus, he is one of the joint recipients of the 2009 Gordon Pask Award for contribution to Agile practice. I invite you to follow his work and learn from his example. On to the substance of Gus’ comment.

Gus, it appears you do not use integration tests to check basic behavior exhaustively. While I try not to use integration tests to check basic behavior at all, I mostly hope to stop programmers from attempting to write exhaustive integration tests that check basic correctness conditions. I wrote in Not Just Slow: Integration Tests are a Vortex of Doom about the vicious cycle I see when teams rely on integration tests to check basic correctness. I encourage them to stop that particular insanity. I would hesitate to use integration tests as even smoke tests for basic correctness, but if I found myself in a situation where I needed to write such tests, I’d do it, then look for ways to render them obsolete.

Also, you mention writing “human-readable acceptance tests”, and I certainly use such tests in my work. When I counsel against using integration tests, I advise it within the context of programmer tests only. While I strongly encourage teams to allow even some of their acceptance tests to check policy or business rule behavior directly and in isolation, I understand and agree that one generally needs to write some acceptance tests as integration tests.

In general, you describe using integration tests quite purposefully, mindfully, and responsibly. I expect no less from a practitioner of your caliber. I would truly enjoy working with you on a project.

Finally, you mention that your integration tests catch system-level issues, such as a broken database schema, mistaken cache integration, and so on. I expect integration tests to find only, or at least mostly, these problems. None of these sound like basic correctness problems.

So Gus, I appreciate you for writing a great description of using integration tests well. I wish we had more examples like this. I truly wish I saw more examples like this. Sadly, I don’t: I see teams trying to check basic correctness issues with plodding, brittle, misleading tests. For those, I stress the need to eliminate integration tests.

January 31, 2010 08:00 testing, agile, design, integrated tests are a scam

Not Just Slow: Integration Tests are a Vortex of Doom

me@jbrains.ca (J. B. Rainsberger) — Thu, 28 Jan 2010 08:00:00 GMT

“Aha! So @jbrains is really against the integration tests just because they are too slow for hourly use”

It reminds me about the Ferrari IT story (XP team, dozens of deployments a year on many continents) that started from getting a big visible counter of a total number of tests and wrote just big amount of any tests first. You need to start somewhere and getting large integration tests is definitely better than nothing. As long as you are prepared to improve the testing practices later. —Artem Marchenko

I agree with this sentiment. I tell the story of my very first attempt at test-first programming¹, how I wrote about 125 tests, many of which fit my definition of “integration test”, and which took 12 minutes to execute. This meant that, on average, I only made 8-12 edits per hour when writing that code. I recognized then, and I still recognize now, that even making only 8-12 edits per hour—4-6 edits per hour towards the end—that I produced better software than I did when I would write code almost continuously for several hours at a time. As much as I disparage those integration tests today, I appreciated them a great deal at the time I wrote them. I find integration tests useful for finding system-level problems, as the first step in fixing a mistake, and if I genuinely can’t write a focused object test, then I will usually write an integration test.

As you say, Artem, I simply don’t stop there.

When I label integration tests a scam, I mean to emphasize the self-replicating nature of integration tests. It starts simply enough: you write a handful of integration tests, which give you a lot of freedom to implement your design in a way that introduces unfortunate dependencies, which makes focused object testing quite difficult. As a result, you will probably resign yourself to writing more integration tests, which do nothing to improve your dependency problems, and the cycle begins again.

Integration tests help cause pain, even though they appear to help reduce pain. Therein lies the scam.

I must acknowledge this: if you started writing tests this week, or this month, or even this year, then you will probably benefit more from writing integration tests than trying to write perfectly focused object tests. I have said and written elsewhere that I believe a programmer needs to write about 1500 to burn into her brain the basic patterns of good tests. Even so, as you write those tests, I want you to remain aware of the cost. Even if you don’t know how to write a good, focused object test, if you want to write more such tests, and especially if you try to write more such tests, then I will have completed the first phase of my mission to eradicate programmer reliance on integration tests to show the basic correctness of their code.

Join us! Turn one integration test into a small suite of focused object tests today. If you don’t yet see how to replace an entire integration test with equivalent focused object tests, then write at least one or two focused object tests along side the integration test. Try it. I promise, you’ll like it.

¹ I use the term test-first programming to refer to test-driven design without the evolutionary design part. With test-first programming, I develop a specific design, then I use tests to help me type it in correctly.

One last comment to my good friend Artem: please don’t put me to sleep with the word “just”!

January 28, 2010 08:00 testing, agile, refactoring, design, integrated tests are a scam

The importance of aligning authority with responsibility

me@jbrains.ca (J. B. Rainsberger) — Mon, 25 Jan 2010 08:00:00 GMT

Some ideas simply click with you. I remember reading about “the alignment of authority and responsibility” for the first time and feeling how much that idea resonated with me.¹ I worked at IBM at the time, on the (then) Net.Commerce project, and after about a year, I began to feel some responsibility for delivering on a project plan without any authority to negotiate scope, priorities, or timing with a stakeholder. I remember clearly staring at a whiteboard that waited for me to put work estimates next to a long list of tasks. I refused to do it.

“I don’t know long it will take,” I told my manager.

“Forget about getting the numbers right. Just give me something to put in the plan.” I knew he supported me, but he needed me to understand the difficulty in his position. He felt a similar powerlessness and asked me to help him work through it. I felt quite sad about it. While his response didn’t help me in the short term, but it opened my eyes to what I now know to call Schedule Chicken. I felt truly powerless in having to make a commitment to a schedule in which I did not believe.

I had had project managers hold me to rough estimates in the past. I had read about the “best practice” called No Hallway Estimates. I felt that Management (that nebulous machine that doesn’t really exist) foisted responsibility on me without the authority I needed to accept and deliver on that responsibility.

I had had enough. I started walking up and down the halls, asking, “Why do you make me guess? I always guess wrong.” Within two years, I left IBM with serious Authority Deficit Disorder. If you find yourself in a position of having to manage people with low authority and high responsibility, then you need to know the Law of Conservation of Authority and Responsibility.

People will correct an imbalance of authority and responsibility, whether you want them to or not, and on their schedule, not yours.

Consider the case of Jane, who has spent two years suffering from Authority Deficit Disorder. If you don’t give her more authority, then she will reach her breaking point, disrupt your organization, and eventually leave. Unfortunately, if you decide to give her more authority, then she will spend about two years demanding authority without responsibility, just to compensate for her two years of Authority Deficit Disorder. She will demand it. Either way, Jane will disrupt your organization, your plans, and probably your life. You will suffer, whether you deserve it or don’t.

So the Law of Conservation of Authority and Responsibility has a corollary.

When a person suffering from Authority Deficit Disorder has reached his breaking point, nothing you do can remedy the situation, except to wait.

Jane spent two years suffering, so she probably needs two years to recover. I understand how powerless and unfair that is, but I don’t find it any less unfair than the treatment Jane endured. This leads me to conclude

When you withhold authority, but insist on responsible behavior, everyone loses.

I don’t know how to solve that problem without simply balancing authority and responsibility better.

¹ I haven’t read it, but found at least one reference to this idea in Harry Chambers and Robert Craft’s No Fear Management.

January 25, 2010 08:00 people, coaching, consulting

Lend your voice and face to Agile India 2010

me@jbrains.ca (J. B. Rainsberger) — Sun, 10 Jan 2010 08:00:00 GMT

I will have the opportunity to address the Agile India 2010 events in Mumbai and Bangalore this month on a simple topic:

What silly thing do you want people on software projects to stop doing in 2010?

I’d love your help. A number of people have already responded on Twitter with the hashtag #StopIt2010, but I invite you to go one step further. Voice your opinion, with video if you can, and audio if you don’t have a camera handy, answering the question I’ve asked above.

Please introduce yourself, say “Hello” to our audience, and give the 20-second version of your rant. You can practise by writing your rant out in 140 characters first, then reading it.

If you can record video, then please simply upload the video somewhere (youtube.com, vimeo.com, dropio.com), add a comment to this entry, and tweet your link with the hashtag #StopIt2010.

If you can only record audio, then please package the audio and a nice, big picture of you, and upload it somewhere, like dropio.com, add a comment to this entry, and tweet your link with the hashtag #StopIt2010.

I’ll pick my favorites and include them in my keynote.

So please… go for it. Rant away. Submit more than one, if you like. You can use this template, if it helps you:

Hello to everyone at Agile India! My name is J. B. Rainsberger and I want programmers to stop insisting that writing the tests after the code is just as good as writing the tests first. Stop it!

Thanks!

January 10, 2010 08:00 agile