Janet wanted a link for her new book “Facilitating a quality assessment” (exact title yet to come) on the test automation pyramid. To our horror, we realized that we have never blogged about it – at least not specifically. It’s in all our books, but not where people who haven’t read our books could read about it. We are correcting that deficiency with this post.
There have been many adaptations of the pyramid model and in our book Agile Testing Condensed, we cover quite a few of them. This post is only about the classic version.
The model
Mike Cohn’s test automation pyramid has helped many teams since the early 2000s. We’ve adjusted it slightly since then to make our intent clear, including the cloud bubble on top to represent that not all regression tests can be automated. Sometimes we need human-centric tests, which include exploratory tests (ET). We consider it a thinking tool to help teams create an effective test automation strategy.
This model helps teams understand that in most contexts, it pays to automate tests at the most granular level of the application as possible, to provide adequate protection against regression failures. Teams that practice test-driven development (TDD) build up a solid base of unit- and component-level tests that help guide code design. In most contexts, teams want to have the biggest proportion of their automated tests at the lowest level of the pyramid. They test functionality for tiny sections of code. Test doubles such as mocks and stubs are used so that tests don’t access external components or databases. These tests run very fast, so they give the team quick feedback. Once team members learn to write tests at this level, they’re easy and quick to write and maintain.
With most applications, testing interactions between different layers of the architecture is required. For example, business logic usually requires interaction with the database. Doing as much automation of this type at the service or API or services level without going through a user interface (UI) is generally the most efficient way. Note: Not all tests at the API level need to touch the database.
Some regression failures only occur when two or more layers of the application are involved. That may require workflow tests through the UI that involve the server, database, and/or an external system. In most contexts, it’s best to minimize the number of end-to-end workflow tests. These tests run slowly, are often the most brittle, and usually require the most maintenance. The cloud at the top of the pyramid includes human-centric activities such as exploratory testing and other tasks that can’t be automated.
The test automation pyramid helps us think of ways to “push tests lower,” maximize the regression tests that are isolated to one part of an application and minimize those that involve multiple parts of the system. It does not imply any specific numbers or percentages of tests at each level.
Getting the whole team involved in formulating a strategy for addressing different automation needs and executing that strategy is key to succeeding with automation. The test automation pyramid is a visual model. It serves as a thinking tool to guide conversations about how your team wants to automate tests.
Collaborate
When your team starts planning a new feature, draw a pyramid (ok, yes, it is really a triangle) on a white board, flip chart, or online collaborative whiteboard such as a Miro board. Discuss what tests might be needed, and what level you want to automate them at. The conversation might evolve into who should do what automation tasks.
We encourage teams to share the responsibility of testing the business rules and higher levels of integration at the API level. Testers should also have visibility into the unit and component tests written by developers. As a team member, it’s important to understand your team’s applications and the inner workings when approaching your automation strategy.
Because automating tests through the UI tends to be more time-consuming, there’s a temptation to hand that off to a separate automation team or have the testers on the team take full responsibility for it. We recommend that the developers, who are good at writing efficient, maintainable code, work together with the testers, who are good at specifying test cases, to automate tests through the UI as well as all other layers above the base level of the pyramid. Having testers and programmers pair or work in ensembles to automate tests saves time. Your team will enjoy better-designed tests whose results you can trust.
Visualize
Remember, like all the other models, the test automation pyramid is a guide. Adapt it if needed to fit your team’s context. Teams that get members in all roles together to ask questions, chat about the answers, draw on the whiteboard, and design experiments have the best success with automation. Visual models help teams talk about why they’re automating tests, what their biggest automation pain points are, how people with different skills could help, and what their next experiments should be. In our experience, a step-by-step approach works best.
