Our definition of “agile testing”
Thanks everyone for your input on our original definition. We’ve changed it slightly based on new conversations and comments.
Definition of agile testing: (by Lisa Crispin and Janet Gregory)
Collaborative testing practices that occur continuously, from inception to delivery and beyond, supporting frequent delivery of value for our customers. Testing activities focus on building quality into the product, using fast feedback loops to validate our understanding. The practices strengthen and support the idea of whole team responsibility for quality.
Where:
- Building quality in, means teams focus on preventing misunderstandings about feature behavior, as well as preventing defects in the code
- Guiding development with concrete examples, means using practices like ATDD (acceptance test driven development, BDD (behaviour driven development) or SBE (specification by example)
- Testing activities include, but are not limited to: conversations to build shared understanding, asking questions to test ideas and assumptions, automating tests, performing exploratory testing, testing for quality attributes like performance, reliability, security, and learning from production usage.
- The whole team uses retrospectives and small experiments to continually improve testing and quality and find what works in their context.
Hi Janet and Lisa,
I like to intention and the spirit of the definition. There are a couple of things I want to raise (hopefully not coming cross as hard to please)
** “using fast feedback loops” – while they might be fast are they useful? Effective might have a place in this part of the definition
** ” Building quality in, means teams focus on preventing misunderstandings about feature behavior, as well as preventing defects in the code”. The use of “preventing” sits uncomfortably with me. I’m inclined to re-interpret your statement as “Building quality in, means teams developing behaviours and practices that reduce misunderstandings about feature behavior, as well as increasing their ability to find defects in the code”
I’ll be sharing this with my team and also the other teams where I work.
Thanks for doing this and sharing.
Paul
Thanks Paul, The beauty of an ever-evolving definition is that we realize it will never be perfect. The hard part is deciding when to release it. Is it good enough for now to get feedback?
That said, I do like the word effective and I hope that everyone realizes that is implied in the idea of fast feedback. As well, a lot of people have an issue with the word ‘preventing’, and perhaps your way is a better way of saying the same thing. I will ponder that for a while.
Paul, I like the “preventing defects in the code” because it stresses the pro-active role of agile testing. Testing is not any more about trying to break the code, but preventing that is breaks.
Inge, may I suggest that testers cannot “prevent defects in the code”. Testers can make suggestions about ways the code could be structured, provide feedback about tricky scenarios, etc, but they do not write the code, and, they cannot compel the developer to write in specific ways. Even if you could compel, would that just remove one type of bug and introduce another? The other question I have is how do you know if you have prevented something? You can do things that may reduce the likelihood of defects but we can never really demonstrate prevention. That something didn’t happen doesn’t demonstrate prevention. Much like bugs, you can really only prove their existence in a system.
Paul, I’ve heard those arguments before. It is one of the reasons we didn’t use the word “tester” as part of the definition. And, it is difficult to prove prevention, but you can measure it but watching the number of defects go down or better yet, the level of happiness from your customers increase.
I also have watched teams in story readiness workshops where the discussion led to say a programmer saying things like “I never thought of that”. To me, that conversation likely prevented a defect in code – but, you’re right. There is no proof.
Except that no tester actually breaks code. We as testers mat cause environmental breakdowns, but this does not necessarily break the code either. It works be better to think of it as revealing areas that the code cannot currently handle well. We reveal issues. In an agile context, I find many of our bugs are subtle use cases that may not be known at the time acceptance criteria are crafted.
@Tim I agree that testers don’t break the code. Exploratory testing is a huge part of a testing strategy in agile teams – to find those subtle issues. Those ‘things’ we didn’t think about. We need many aspects but it starts with asking questions to get misunderstandings out of the way.
How about
“increasing understanding”, rather than “preventing misunderstandings”. I like
I like that, though we also mention “conversations to build shared understanding”, maybe we can tighten all that up a bit. Thanks!
I generally liked the early attempt t the definition. I agree with Paul that preventing defects isn’t something testers really can control. We are part of the solution but unless we become hybrid developer/test engineers and have our hands in the production code, I find that prevention just doesn’t seem like its within our power. We can certainly influence up front and as something is developed and that, on its own might prevent most defects, but given our Server Farm, Virtual Machine Driven, Cloud riding, DevOps, Continuos Delivery environments in the most leading edge of companies, the myth of complete testing, even in agile still remains.
There will always be situations that no matter how proactive we try to be, no matter how much testing for scale or security we might do, that we will run into issues that we just could not have foreseen. This particularly tails in the -ility space. Security, Performance being the chief areas where I feel even the best skilled, best trained, best teams with agile testing being practiced will sometimes not be able to overcome, the operations side of things. Even our wildest optimistic estimates of how hard an app might get pushed could get blown out of the water in some contexts. And hence, this is why I dislike ‘prevent defects’. I prefer instead to consider our role as revealing product and customer use cases. (Being that the two are not always the same).
Agreed, Tim. Lots of critical quality attributes like security, performance, accessibility, reliability can get overlooked in the drive for functionality and user experience. We said specifically “preventing defects in the code”, but we really want focus to be on “building quality in” – meaning all aspects of quality.
This is a great definition. Couple of questions:
– The second sentence “Testing activities focus on …” seems to be missing “and”, was there something removed?
– “Guiding development with concrete examples” is explained, but isn’t used in the definition. Could that be added to the second sentence, so it would read:
“Testing activities focus on building quality into the product, using fast feedback loops to validate our understanding, and guiding development with concrete examples.”