I have already submitted one session to Agile 2010, and I don’t know what else to submit for the year. I’d like to leave one slot open for a potential collaboration (hint, hint), but as for a solo session, what do you think I could present or facilitate that you’d spend 90 minutes of your precious time attending? I’d prefer to do something new, which means no “Integration Tests Are A Scam” and no “XP: My Greatest Misses”, but that doesn’t mean that I couldn’t run a session on this topics.

You could help me by outlining what you might see in a session description in the Agile 2010 program that would attract you over the other sessions. Perhaps I can deliver that very session!

Thanks for your help.

Gus Power commented about the way he uses integration tests in his work.

Interesting series of articles & comments. I also read Steve Freeman’s article in response to the same topic. It’s got me thinking about how we work and I thought I’d take the time to describe it here.

You define an integration test as “… any test whose result (pass or fail) depends on the correctness of the implementation of more than one piece of non-trivial behavior.” We have many such components that exhibit such non-trivial behaviour in the products we create, many of which are not developed by us. And we have integration tests to verify they work. I’m not just talking about 3rd party libraries and frameworks here, I’m referring to the whole system: caching layers. load balancers, DNS servers, CDNs, virtualization etc. When we build software it only becomes a product or service for our users when it has been deployed into a suitable environment; an environment that typically contains more than just the software we have written and packaged. Since our users’ experience and perception of quality result from their interaction with a deployed instance of the whole system, not just their interaction with the software at a unit level, we have come to value end-to-end integration testing. I believe there’s merit in testing these components in symphony and will attempt to clarify what kind of integration testing I’m talking about.

For a given piece of functionality we write an executable acceptance test in human readable form (for web projects we typically use some domain-specific extensions to selenium, for services we have used FIT and it’s ilk, sometimes we roll our own if there’s nothing expressive enough available). We run it against a deployed version of the application (usually local though not always) which typically has a running web/application server and database. The test fails. We determine what endpoint needs to be created/enhanced and then we switch context down into unit-test land. A typical scenario would involve enhancing a unit test for the url mappings, adding one for the controller, then one for any additional service, domain object etc. When we’re happy and have tested and designed each of the required units we jump back up a level and get our acceptance test to progress further. The customer steers the development effort as he sees vertical ‘slices’ of functionality emerge. The acceptance test is added to a suite for that functional area. The continuous build system will then execute that test against a fully deployed (but scaled down) replica of the production environment, with hardware load balancer, vlans, multiple nodes (session affinity) and so forth. Any additional environmental monitoring (e.g. nagios alerting) is also done as part of this development effort and is deployed into the test environment along with the updated code.

Setting up the infrastructure to do this kind of testing takes investment, both initial and ongoing. The continuous build needs to be highly ‘parallelized’ so you get feedback from a checkin in 10 mins or less (we’re heavy users of virtualization, usually VMWare or OpenVZ). The individual acceptance test suites need to be kept small enough to run quickly before check-in.

Benefits of this approach
* The continuous context-switch between acceptance test and unit test is key to our staying focused on delivering what the customer actually wants.
* The customer has multiple feedback points that he can learn from and use to steer the development effort.
* It confirms that the whole system works together – networking, DNS, load balancing, automated deployment, session handling, database replication etc.
* We create additional ‘non-functional’ acceptance tests that automatically exercise other aspects of the system such as fail-over and recovery.
* Upgrades to parts of the system (switches, load balancers, web caches, library versions, database server versions etc.) can be tested in a known and controlled way.

We’ve caught a number of integration-related issues using this approach (a few examples: broken database failover due to missing primary keys, captcha validation not working due to a web cache not behaving correctly, data not persisting because one database server had the wrong locale) and stopped them before they have reached our users. We have used the feedback as a basis for improving our products and their delivery at a system level.

OK this reply has now become far too long :-/ It would of course be good to discuss this in person sometime :)

—Gus Power

Thanks for the substantial comment, Gus. For those who don’t know Gus, he is one of the joint recipients of the 2009 Gordon Pask Award for contribution to Agile practice. I invite you to follow his work and learn from his example. On to the substance of Gus’ comment.

Gus, it appears you do not use integration tests to check basic behavior exhaustively. While I try not to use integration tests to check basic behavior at all, I mostly hope to stop programmers from attempting to write exhaustive integration tests that check basic correctness conditions. I wrote in Not Just Slow: Integration Tests are a Vortex of Doom about the vicious cycle I see when teams rely on integration tests to check basic correctness. I encourage them to stop that particular insanity. I would hesitate to use integration tests as even smoke tests for basic correctness, but if I found myself in a situation where I needed to write such tests, I’d do it, then look for ways to render them obsolete.

Also, you mention writing “human-readable acceptance tests”, and I certainly use such tests in my work. When I counsel against using integration tests, I advise it within the context of programmer tests only. While I strongly encourage teams to allow even some of their acceptance tests to check policy or business rule behavior directly and in isolation, I understand and agree that one generally needs to write some acceptance tests as integration tests.

In general, you describe using integration tests quite purposefully, mindfully, and responsibly. I expect no less from a practitioner of your caliber. I would truly enjoy working with you on a project.

Finally, you mention that your integration tests catch system-level issues, such as a broken database schema, mistaken cache integration, and so on. I expect integration tests to find only, or at least mostly, these problems. None of these sound like basic correctness problems.

So Gus, I appreciate you for writing a great description of using integration tests well. I wish we had more examples like this. I truly wish I saw more examples like this. Sadly, I don’t: I see teams trying to check basic correctness issues with plodding, brittle, misleading tests. For those, I stress the need to eliminate integration tests.

“Aha! So @jbrains is really against the integration tests just because they are too slow for hourly use”

It reminds me about the Ferrari IT story (XP team, dozens of deployments a year on many continents) that started from getting a big visible counter of a total number of tests and wrote just big amount of any tests first. You need to start somewhere and getting large integration tests is definitely better than nothing. As long as you are prepared to improve the testing practices later. —Artem Marchenko

I agree with this sentiment. I tell the story of my very first attempt at test-first programming¹, how I wrote about 125 tests, many of which fit my definition of “integration test”, and which took 12 minutes to execute. This meant that, on average, I only made 8-12 edits per hour when writing that code. I recognized then, and I still recognize now, that even making only 8-12 edits per hour—4-6 edits per hour towards the end—that I produced better software than I did when I would write code almost continuously for several hours at a time. As much as I disparage those integration tests today, I appreciated them a great deal at the time I wrote them. I find integration tests useful for finding system-level problems, as the first step in fixing a mistake, and if I genuinely can’t write a focused object test, then I will usually write an integration test.

As you say, Artem, I simply don’t stop there.

When I label integration tests a scam, I mean to emphasize the self-replicating nature of integration tests. It starts simply enough: you write a handful of integration tests, which give you a lot of freedom to implement your design in a way that introduces unfortunate dependencies, which makes focused object testing quite difficult. As a result, you will probably resign yourself to writing more integration tests, which do nothing to improve your dependency problems, and the cycle begins again.

Integration tests help cause pain, even though they appear to help reduce pain. Therein lies the scam.

I must acknowledge this: if you started writing tests this week, or this month, or even this year, then you will probably benefit more from writing integration tests than trying to write perfectly focused object tests. I have said and written elsewhere that I believe a programmer needs to write about 1500 to burn into her brain the basic patterns of good tests. Even so, as you write those tests, I want you to remain aware of the cost. Even if you don’t know how to write a good, focused object test, if you want to write more such tests, and especially if you try to write more such tests, then I will have completed the first phase of my mission to eradicate programmer reliance on integration tests to show the basic correctness of their code.

Join us! Turn one integration test into a small suite of focused object tests today. If you don’t yet see how to replace an entire integration test with equivalent focused object tests, then write at least one or two focused object tests along side the integration test. Try it. I promise, you’ll like it.

¹ I use the term test-first programming to refer to test-driven design without the evolutionary design part. With test-first programming, I develop a specific design, then I use tests to help me type it in correctly.

One last comment to my good friend Artem: please don’t put me to sleep with the word “just”!

Some ideas simply click with you. I remember reading about “the alignment of authority and responsibility” for the first time and feeling how much that idea resonated with me.¹ I worked at IBM at the time, on the (then) Net.Commerce project, and after about a year, I began to feel some responsibility for delivering on a project plan without any authority to negotiate scope, priorities, or timing with a stakeholder. I remember clearly staring at a whiteboard that waited for me to put work estimates next to a long list of tasks. I refused to do it.

“I don’t know long it will take,” I told my manager.

“Forget about getting the numbers right. Just give me something to put in the plan.” I knew he supported me, but he needed me to understand the difficulty in his position. He felt a similar powerlessness and asked me to help him work through it. I felt quite sad about it. While his response didn’t help me in the short term, but it opened my eyes to what I now know to call Schedule Chicken. I felt truly powerless in having to make a commitment to a schedule in which I did not believe.

I had had project managers hold me to rough estimates in the past. I had read about the “best practice” called No Hallway Estimates. I felt that Management (that nebulous machine that doesn’t really exist) foisted responsibility on me without the authority I needed to accept and deliver on that responsibility.

I had had enough. I started walking up and down the halls, asking, “Why do you make me guess? I always guess wrong.” Within two years, I left IBM with serious Authority Deficit Disorder. If you find yourself in a position of having to manage people with low authority and high responsibility, then you need to know the Law of Conservation of Authority and Responsibility.

People will correct an imbalance of authority and responsibility, whether you want them to or not, and on their schedule, not yours.

Consider the case of Jane, who has spent two years suffering from Authority Deficit Disorder. If you don’t give her more authority, then she will reach her breaking point, disrupt your organization, and eventually leave. Unfortunately, if you decide to give her more authority, then she will spend about two years demanding authority without responsibility, just to compensate for her two years of Authority Deficit Disorder. She will demand it. Either way, Jane will disrupt your organization, your plans, and probably your life. You will suffer, whether you deserve it or don’t.

So the Law of Conservation of Authority and Responsibility has a corollary.

When a person suffering from Authority Deficit Disorder has reached his breaking point, nothing you do can remedy the situation, except to wait.

Jane spent two years suffering, so she probably needs two years to recover. I understand how powerless and unfair that is, but I don’t find it any less unfair than the treatment Jane endured. This leads me to conclude

When you withhold authority, but insist on responsible behavior, everyone loses.

I don’t know how to solve that problem without simply balancing authority and responsibility better.

¹ I haven’t read it, but found at least one reference to this idea in Harry Chambers and Robert Craft’s No Fear Management.

I will have the opportunity to address the Agile India 2010 events in Mumbai and Bangalore this month on a simple topic:

What silly thing do you want people on software projects to stop doing in 2010?

I’d love your help. A number of people have already responded on Twitter with the hashtag #StopIt2010, but I invite you to go one step further. Voice your opinion, with video if you can, and audio if you don’t have a camera handy, answering the question I’ve asked above.

Please introduce yourself, say “Hello” to our audience, and give the 20-second version of your rant. You can practise by writing your rant out in 140 characters first, then reading it.

If you can record video, then please simply upload the video somewhere (youtube.com, vimeo.com, dropio.com), add a comment to this entry, and tweet your link with the hashtag #StopIt2010.

If you can only record audio, then please package the audio and a nice, big picture of you, and upload it somewhere, like dropio.com, add a comment to this entry, and tweet your link with the hashtag #StopIt2010.

I’ll pick my favorites and include them in my keynote.

So please… go for it. Rant away. Submit more than one, if you like. You can use this template, if it helps you:

Hello to everyone at Agile India! My name is J. B. Rainsberger and I want programmers to stop insisting that writing the tests after the code is just as good as writing the tests first. Stop it!

Thanks!