We have used the agile testing quadrants as a thinking tool and planning guide since Brian Marick first wrote about them as an agile testing matrix in 2003. We published our own adaptation (with Brian’s blessing) in our first Agile Testing book and continued to evolve them as we learned new ways of testing. Other teams have adapted them to fit their own context. You can download Chapter 8 from our More Agile Testing book to see some of these adaptations. In 2019, while writing our newest book Agile Testing Condensed, we added observability and testing in production into the top right quadrant, business-facing tests that critique the product.
Recently, we started to think about how the quadrants model can help teams succeed with continuous delivery (CD) and deployment (also CD). We’ve explored some ways this thinking tool can spark and guide conversations that help teams succeed in their journey towards continuous delivery/deployment.
This is a two-part series. Here in Part 1, we discuss using the agile testing quadrants in the context of working towards continuous delivery. Part 2 is about using the quadrants to plan your delivery pipeline.
Build a common language
In today’s complex world, we can easily be divided by the “lingo” we use. We’ll define a couple of terms so we’re all on the same page and start with the popular term, “DevOps”.
Agile transformations “should” break down all silos and create cross-functional teams. Yet too often, the operations specialists are left out. DevOps is a label for a quality-oriented culture where the people who build the software also take care of it in production. They focus on building small, incremental, low-risk changes, which they release frequently. Rather than throwing a release candidate over the wall to an Operations (Ops) team, people in all specialties share responsibility for building the software and taking care of it in production.
More and more teams are moving into a continuous delivery world where small changes are deployed to production safely and sustainably. Jez Humble and Dave Farley coined the term “continuous delivery” to describe code in a deployable state – all the time, even if many developers are continually making changes.
Our continuous world
Teams adopting a DevOps mindset working towards continuous delivery, configure deployment pipelines which include many testing stages – both human-centric and automated. They use release feature toggles or feature flags to hide changes in production while they complete testing. They release when they feel confident, and they continually watch production to look for anomalies and errors.
This continuous delivery world has begun to make sense for many of us, yet we see many teams scratching their collective heads. “It was hard enough to fit testing activities into two-week release cycles. How the heck do we get enough testing done when we release twice a week, every day, multiple times per day?” This is where the agile testing quadrants can ease the transition.
The quadrants are a thinking tool. Your team can use them to create a common language to make conversations easier. It helps with planning at all levels. Notice that we haven’t said anything about “only” testers using the quadrants. It is a tool that enables the whole team to take responsibility for planning and executing testing activities.
Applying the quadrants to plan testing in DevOps
Let’s take a tour of each quadrant to show how it helps guide conversations about risk and testing activities in a DevOps culture. The quadrants themselves don’t imply any sequence; they’re a way to categorize the activities.
In the upper left quadrant (yellow), we have business-facing tests that guide development. Activities from this quadrant help achieve the business goals and make sure teams deliver the right value to customers while mitigating risks from a business perspective. Naturally, there are “functional” requirements, such as desired application behavior, which can be turned into executable tests at the API level or workflow tests through the UI to guide development. Teams also need to consider business risks – for example, what if your app depends on third-party services? As teams discuss these activities around business-facing quality, they can plan what instrumentation they need to add to their code to capture data for monitoring and observability.
The lower left blue quadrant, technology-facing tests that guide development, reminds teams to talk about the risks they face from a technology point of view before they start writing new code. Building maintainable, operable, testable code helps manage these risks and lets teams catch errors as early as possible. The team can incorporate telemetry as they code.
The green top right quadrant, business facing tests that critique the product, is again more concerned with business risks. Teams may choose to deploy to pre-production environments for exploratory testing to build confidence for production deployment. Even with this testing, there is no way to replicate every possible scenario that customers will do with your application. So, teams also need to keep an eye on production usage with monitoring dashboards and observability tools, so they can find and fix issues before customers are affected by them. Your discussions around this quadrant may include what release feature toggles are needed to hide new changes from customers until you’ve done some testing in production.
The lower right-hand orange quadrant, technology-facing tests that critique the product, includes many quality attributes. Think of these as constraints on your system. What are your biggest risks? For example, if rolling out a new feature causes an issue, how to you handle recoverability? Is your process robust enough that you can deploy a fix in a short time, or do you have to quickly roll back the latest deployment? You can talk about the infrastructure needed to safely conduct “chaos engineering” to test system resiliency.
These are just a few examples of how teams can use the quadrants can guide conversations to plan testing activities to help them successfully adopt continuous delivery.
In Part 2, we look at how teams can use the quadrants to build their continuous delivery pipeline and other infrastructure needed to succeed with continuous delivery.
Thank you!!1
Thank you!!1