Hi, I’m a Quality Engineer at my organization and I recently watched your video about integration tests being a scam.1 We’re rearchitecting a lot of our system and exploring ways to better automate our testing to ensure we can deploy faster and with more confidence and I’m exploring ways the quality team can support this. How do you see a quality team working in a world where integration tests aren’t helpful in reducing bugs/mistakes? If the focus is on low level unit tests which are often better facilitated by developers, what can automation engineers do to support the product?

I’d like to start by clarifying a key point: I find integrated tests helpful in reducing bugs/mistakes, but I rely on them less than other people do. I tend to avoid mistakes with mostly microtests and integrated tests at the boundaries. I tend to use integrated tests to explore unexpected problems, then I tend to replace those integrated tests with microtests as I pinpoint the source of the problem.

All this means that if an integrated test finds a defect, then we ask, “Why didn’t we find this defect with a unit test?” And we ask this question in an open way, not in a blaming way: most of the time, we notice that we missed a unit test that we would have written if we had had all the time in the world, but sometimes we find a defect that nobody saw coming, and that’s when we feel grateful for the integrated test.

I say this not as a form of “well, actually…”, but because explaining that provides some context that makes it more convenient to answer the question.

How do I prefer to work with automation engineers? First, let me describe how I prefer to work with human testers, then how the automation engineers act as a bridge between programmers and testers.

Behold the Human Tester

First, imagine the role of the human tester. In The Bad Old Days, a programmer would write code, then throw it over a wall where it would land on a human tester. The human tester would then run some “tests”, probably by following a Test Script or a Test Plan, to do what we now call “routine checking” in the Michael Bolton2 sense. Some folks saw this testing role as waste, so we decided one day that programmers ought to become testers and we called it “Test-First Programming”, which later became known as “Test-Driven Development”. From that point forward, the programmer took primary responsibility for routine checking, leaving the human tester to do… what, exactly?

The human tester no longer needed to do routine checking, because the programmer was doing it. This meant that the human tester could turn their attention to more interesting risks in the system. We use names such as “Exploratory Testing” and “Soap Opera Testing” to describe a kind of extraordinary testing, the kind that goes beyond the kinds of routine checking that “even a programmer” could be trusted to do. As a programmer, what I then need from the human tester is no longer to automate routine checks, but instead to explore the product, to challenge assumptions, and to subvert expectations. Instead of playing the role of parent cleaning the child’s face with a wet cloth, the human tester now has the programmer’s back in a fight (presumably against the defects in their system?). The programmer feels comfortable doing the routine checking, confident that the human tester covers their blind spots. They communicate; they compensate for each other’s weaknesses; they complement each other.

Enter the Automation Engineer

Now we come to how I ask the automating engineer to help: establish a pipeline for automating the human tester’s work when they uncover a test that the programmer would like to adopt as a routine check. The automation engineer helps the human tester avoid falling back into the trap of mindlessly running routine checks. True: these new routine checks are more interesting and more powerful than the previous ones, but the purpose of automation is to do repetitive things so that humans are free to become even more creative.3 The automation engineer plays a pivotal role as an agent of the human tester, relieving them of the burden of executing the same checks over and over again. The human tester explores the system and as they find interesting behavior, the resulting tests transform gradually into checks. The automation engineer automates those checks, so that the programmer can benefit from them as an ever-improving pool of change detectors.

Moreover, I’d like the automation engineer and the programmer to work together to transform bigger-scope (integrated) tests into smaller-scope (micro-) tests, because the automation engineer understands enough software design and testing to bridge the communication gap. The programmer doesn’t have to become an expert tester, nor does the tester have to become an expert programmer, but rather the automation engineer understands both groups and facilitates the transition from larger-scope extraordinary tests to smaller-scope routine checks.4

Moreover moreover, the programmer is allowed to focus on unit tests, so they write more of them, which strengthens those tests as a mechanism for finding defects and supporting refactoring. Consequently, the tester is less-often distracted by silly mistakes that the programmers really could have found or avoided on their own. The tester has more time and energy to devote to finding even more interesting problems before the end users do. In this way, the testers become free to act as agents of the end user or customer. They can think more about the system as a product from the end user’s point of view, rather than being relegated to treating the system as a bunch of bug-riddled software components in need of constant attention. Over the longer term, the tester develops a greater understanding of how the product might satisfy the customer—and even delight them!

Everyone wins. Let’s go!

  1. I strongly strongly strongly prefer people to watch this version of the talk to any of the previous versions of the talk. And if you’ve seen one of the older versions, then please watch this next.↩︎

  2. No, not that one, and not that one, either, but this one.↩︎

  3. Something something generative AI something something.↩︎

  4. The mediator between Programmer and Tester may be the Automation Engineer.↩︎