Automated Testing 2: The Test Pyramid

A colleague of mine, Rob Capretto, introduced the idea of a Test Pyramid. Its like a Food Pyramid – everything has its place in moderation, but to stay healthy, there are certain types of tests you should have more of, and others you need only in small amounts. Here’s my pyramid:

testpyramid.png

Notice that

  • The majority of tests are mock-based unit tests. In a perfect world, these would be all you need. However, due to both gaps in coverage and subtle inter-object interactions, in practice its wise to put your pieces together sometimes in an integration test.
  • We have some subsystem integration tests that test parts of the system.
  • We have external integration tests that test software & services created by other people/providers that we rely on, and your infacing to them. How important these are depends on how much “surface area” your system has – if its largely glue, these might predominate.
  • We have a few whole system functional tests that test the fully operational system as a user might use it. These should be very few in number, and just serve as sanity checks that when everything is combined, there aren’t any weird interactions or incompatiblities that prevent it working as intended.

Two tests Good, Four Tests Better…. NOT!

A key point is that, just like in a Food Pyramid, you can easily have too many functional, integration and even unit tests if you’re not careful. Not only are they expensive to create, but having too many test quickly become a maintenance burden when you refactor software. The longer your projects runs for, the more pain you will feel from maintaing tests, because tests tend to multiply inevitably over the lifetime of software, and they become brittle, decayed and obtuse if uncared for, just like application code.

In fact, you have to be sparing in the creation of tests. More testing is not automatically better.

Its really the same principle as application code: you want the most effective code in the least number of lines. For tests, “effective” means detecting bugs and undesired change, whilst minimally obstructing desired change.

(The title is a reference to a favorite line of of mine from George Orwell’s novel Animal Farm, implying overly simplistic dogma: “four legs good, two legs bad”)

Automated Tests Make Brittle Codebases

A provocative subtitle, but it’s literally true! If you take heavily automated-tested code and change almost anything, the test suite will break somewhere. Of course, the breakage is intentional, but dont forget that autoated testing is a special kind of deliberate & detectable brittleness added to code. That brittleness protects against entropy, but it can also obstruct you when you add desired change to the codebase. Ive experienced cases where a small change applied into a large codebase can require ten times more effort to fix breaking existing tests than it does to add the new change and its unit test.

About these ads

6 Comments

  1. hiremaga said,

    April 12, 2008 at 8:29 pm

    “Automated Tests Make Brittle Codebases”

    It’s certainly provocative, but also somewhat misguided IMHO but I’ll try to avoid digressing into a philosophical discussion about trees falling in forests.

    The experience you describe says more about the quality of the tests themselves rather than the affect their mere presence has on a software system. Perhaps one lesson here is that poorly written automated tests can be more brittle than well written ones.

    If unit tests are well written they should be focused and orthogonal only one should ever fail at a time.

    If functional tests are well written, in addition to other attributes, they should be agnostic of implementation specifics meaning none will fail due to a mere refactor.

    If many well written functional tests fail it might be a sign you’re trying to refactor a bicycle into a Ferrari (analogy plagiarised from http://www.eaves.org/jon/work/agile.shtml)

    Now there’s a series of provocative statements :)

  2. benhutchison said,

    April 12, 2008 at 9:54 pm

    “It’s certainly provocative, but also somewhat misguided IMHO but I’ll try to avoid digressing into a philosophical discussion about trees falling in forests.”

    Sorry, Abhi, you lost me with this bit??

  3. hiremaga said,

    April 13, 2008 at 7:46 pm

    I suppose I did what I said I’d try to avoid :)

    I was drawing a comparison with this: http://en.wikipedia.org/wiki/If_a_tree_falls_in_a_forest

    I meant that the tests don’t make a codebase brittle; they may allow its sensation but its reality is independent of the presence of tests.

    I do agree that tests can themselves be brittle and I really like the 4 qualities you explored in your last post.

  4. benhutchison said,

    April 13, 2008 at 8:37 pm

    You make a good point that has clarified my thinking.

    Closer to what I really meant is: “Tests are intentionally brittle code. Adding them to a codebase will increase the average brittleness of the codebase as a whole (ie app + test) ”

    I’ll update the post sometime.

  5. August 13, 2009 at 7:26 am

    Brittle tests are not a sign of a brittle code base. They are a symptom of badly written tests. If you are spending 10 times as much effort changing breaking tests then your tests are too closely coupled to the production code. You shouldn’t have to make the same change in more than a few tests or test utility methods.

    You can use test refactorings to get from brittle tests to robust tests. I describe how to do this (and how to know that you should do it) on my web site http://xunitpatterns.com and my book. (xUnit Test Patterns – Refactoring Test Code)

  6. April 1, 2011 at 11:44 pm

    [...] mates with your bugs. But you can’t live with only unit tests can you? That’s where the test pyramid metaphor comes in. Here’s [...]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: