A student in one of Janet’s classes asked her for a definition of agile testing. He wanted a nice clean definition, which he didn’t get from our 3-day class. Janet went through both of the books we authored, Agile Testing and More Agile Testing, and found that we do not have a succinct definition in either one. The closest she could find in was in this blog post. http://janetgregory.ca/agile-testing-is-not-a-methodology/.
We decided to go on a quest and posed this question to both the Linked-in agile testing group and the long-standing agile-testing yahoo group — “What do you think the definition of agile testing is?”
We found that many people tried to define agile, or used the word agile in the definition. Some people disagreed with the term altogether, but the word “agile” itself is an adjective, so we think agile testing is accurate. We also didn’t want to focus on the tester role, but the testing activity itself.
Based on all the conversations and our own thoughts, here is the definition we came up with:
We hope that this resonates with you, but if not, please leave comments here, or email us. We view this as an opening, a draft if you will. We are open to changing and adapting this definition.
We appreciate all the people who contributed already. Some ideas were helpful ideas, some not so helpful, but all contributed to our thinking. Here are some of the comments and definitions contributed by group members. You can also go to either group to read the entire conversations.
- Collaborative and continuous activity geared to enable error free fast release of products that matter
- Why not testing in an agile context, that is testing in an environment with rapid iterations and continuous Improvement, responding to change and pressure
- One person pointed us to Elisabeth Hendrickson’s Agile Testing Overview, which we encourage everyone to read, but again – it is 21 pages – not a concise definition.
- Effective testing with short turnaround time providing quick feedback to support iterative based deliverables
- What about, Agile Testing is a way of working from day 1 and continuously till it’s delivered into production
- Agile testing focuses directly on business value and the quality customers require. In agile testing, a tester is a member of an agile development team, which, as a team, is responsible for quality.
- Test from a customer perspective and add business value. For me this is essential since this is the goal of the product. In terms of Gojko Adzic’s Software Quality Pyramid – Usable, Useful and Successful Software are often not topics considered to test. To me an ‘agile tester’ should always be aware of all these levels and at least inform the team on how to test these.
- Strengthening the team responsibility for quality.
Someone mentioned the testing manifesto from Karen Greaves and Samantha Laing (Growing Agile)
- Agile Testing is an integral part of Agile Software Development rather than a separate activity.
- Agile Testing is anticipating bug instead of waiting for them to happen!
- Collaboration/agile brings to tester, the fact they are involved all along the project and not just at the end (or a bit at the beginning then the end).
- The more you question the user story and you help the team to surface acceptance criteria, the better it is in the project. The later you find a bug, the more expensive it is for the project.
- Agile testing is testing that can keep pace with short-cycle development.
- Testing practices that support the values and practices of the agile manifesto.
Thanks to these folks who took the question as we intended and tried to help us with our quest. We apologize if we missed anyone. Augusto Evangelisti, Tim Western, Gaurangi (Surve) Rawat, Richard Cowling, Vivek Sharma, Wim Heemskerk, Eddy Bruin, Steven Gordon, Lionel Champalaune, George Dinwiddie, Adrian Howard
I am struggling with this definition. Maybe it would help to define what “testing” is first?
Maybe I have just been on agile teams for so long that I forget the differences between “agile testing” and “testing”.
It seems that I could remove “Agile” and this sentence is still true: “testing includes (but is not limited to) these testing activities: guiding development with concrete examples, asking questions to test ideas and assumptions, automating tests, performing exploratory testing, testing for quality attributes like performance, reliability, security.”
Again, I am just not sure how we are differentiating “Agile Testing” from whatever other testing is out there.
I guess to differentiate, it would be testing in a phased and type project vs in an agile team. One major difference would be that I don’t think have never seen tests written first guiding development in a phased and gated project. Many of the testing activities are the same, but testing is usually seen as a phase at the end or perhaps at intervals throughout the project, rather than activities that happen continuously in parallel with other activities such a programming or refining requirements.
Does that help?
Yes, this helps, but I don’t feel this is well-defined in the original definition given. 🙂
Based on this information could the definition of Agile Testing be better defined as:
Agile testing is testing in an agile environment where we place testing activities in parallel with other development activities such as programming and refining requirements.
?
I like that although we tried to avoid using the word agile in defining the term. Your definition is nice and simple, although it misses the whole team approach to quality and in my experience that is something that is different, although – not always.
Agreed. The hard part of this is encapsulating all the essential aspects of testing in agile into a concise definition. I like yours, Kevin, but as Janet said, it leaves out some key components. But the good thing is we are all thinking and talking about it – that helps me improve as a tester!
What if rather than using the word “agile” in the definition you said something like,, “Agile testing is testing that occurs in a lean software development environment where we place testing activities in parallel with other development activities such as programming and refining requirements?”
Great point, I also struggle with this definition. In my opinion you shouldn’t be separating the two.
Testing is testing. It’s about WHAT you do, which is essentially checking whether the system meets expectations. Agile is HOW you do it. As a process rather than an event. Included rather than excluded. Flexible rather than strictly following the plan. Agile is a mindset, not a methodology.
So agile testing is a more adaptable and responsive way of ensuring the system meets customer expectations (both internal and external).
Karin, I agree with your statement that agile is a mindset. And if testing is checking whether the system meets customer expectations, I think an important part of that is making sure the team understands the expectations before they deliver the feature. I think that’s an area where testers contribute a lot, because we think of questions and quality attributes that others might not think of. Over the years I’ve been on great teams that could write robust code supported with automated tests – but often we delivered something that wasn’t what the customer wanted at all!
That’s why I don’t believe in automation as part of the test effort (and I’m not talking about TDD as that’s a developer function not a test function). It’s the opposite of agile for so many reasons.
Hi Karin, I’d like to hear more of your views on automation. IME we need to support ourselves with good automated regression test coverage as well as automated builds and deploys. Get those fast feedback loops in place. But I think all of those automation activities (other than TDD which as you say is really for code design) should be a whole-team effort. On my current team, the devs do all the automation, and in spite of tens of thousands of regression tests, we don’t have good enough coverage. Tester input is usually needed, from what I’ve seen.
Hi Lisa,
I’ve decided to write a short blog on how I do testing (after I decided a few years ago that I don’t want to be a tester anymore because no-one ever seems to listen to my opinions ;-)) and cover my main view of why I don’t automate as part of testing.
https://medium.com/@funficient/what-is-agile-testing-exactly-1dabb7f09e8
It’s a very short summary of my view, but if you’re interested in more let me know and I can elaborate.
Karin, thanks for sharing your post, I enjoyed it. There are so many different contexts for agile testing and I love learning how other teams approach it. I like how you use personas and think of the questions to ask them. You focus on value to customers, which is so important and can be difficult to do.
My teams have found a lot of value over the years in writing acceptance / story tests before and during coding as executable tests. One thing this allows is to leverage automation to help with exploratory testing, for example, being able to test lots of different inputs by cranking through them via automation. Frameworks like FitNesse where we can put the values in a table format are helpful for this. We don’t keep all these test cases for regression, but they’re helpful for learning while we are coding. And, guiding development with these business-facing tests helps the team gain a shared understanding of each feature and story, and saves time in the long run because we’re more likely to deliver what was needed.
Nice discussion.
Feeling the same need to define what agile testing is (with respect to testing) I had a session with my colleagues during last weekend hackathon. We chose to define characteristics of the agile tester. It became a manifest that states: “I am a agile tester because….” Reading this blog I realise that these combines the “what is testing” and “the how do I do the testing”. It is written in dutch, and do not have the time to translate is today, but I contains the mix of both: How do I contribute to the team so that quality solutions that have value are being developed and released.
We plan to use it as a kind of self-assessment and make people think about the way they do their testing. I will check the items with the things mentioned in this blog.
Struggeling with the question whether there is more value in the definition of testing or the definition what a tester has to do to make it effective in agile setting. Any ideas?
Hi Derk-Jan,
Janet and I started our first book Agile Testing with “10 Principles for Agile Testers”. DZone published it. We talk about the agile testing mind-set, and the principles we think are important for an agile tester. We hadn’t tried to define “agile testing” per se before. Whether we try to define testing or what makes a tester valuable, we are generating more conversations and, for me at least, more insights into keys for successful testing and improving the way we develop and deliver software.
— Lisa
I am okay with any attempt to define what Agile testing is, however Kevin’s comment makes a good point Testing is testing, with or without having Agile attached to it.
Testing is testing – absolutely. It’s really about how you approach it. It’s why Lisa and I didn’t attempt to try to define “testing” in either of our books – it is a subject on it’s own. However, I find that many define testing as testing the software, or perhaps testing the product (better imo), whereas I think that questioning ideas is also a form of testing. We are testing people’s assumption before we start to code and that is what guiding development with examples is all about.
I’ve always had an issue with labels, even “agile”. They become overloaded with meanings to the point they may not mean anything anymore. Rather than “agile development”, I’d rather we just focused on good ways of delivering quality software frequently, and continually improve how we work.
But over the years I’ve become convinced that humans just seem to need labels. It’s important to me that we define them so that we share an understanding of what each one means. But, we can all still have different perspectives.
When I worked in waterfall, I involved myself in each project from the start. You can get a lot of value from testing requirements documents! And before I learned about waterfall, my first dev team worked in much the same way as “agile” teams now. So I agree – let’s do the best job we can with testing activities – and in my experience, that works best when the whole team takes responsibility for them.
At what point will Software Quality Vs Testing get separately identified in Agile methodology?
we build in quality partly by testing early, but testing does not equal quality, so I’m a bit confused by your question.
It’s a very good effort and much needed one. I see where it’s going and encouraging to see so many contributions already.
I like to see when the definition brings out more “Build Quality in” vs “test quality in” approach and how agile manifesto can be leveraged. Also, I like this definition effort to embed to an extent possible the level and the relevance of automation in it. Good luck and will stay tuned.
Thank you, Shiva! That’s an interesting point about automation. I need to think on that.
My two cents here,
This is an awesome one. “Build Quality in” vs “test quality in” . This is what should be the first gold and bold line in the manifesto. I feel we are trying to fight against a word “Testing” and trying to pull this word in the agile world. We will fight for ever if we do this because the word “Testing” resembles more like “something that happens after the fact”. We dont want something that happens after the fact in Agile. Thats our primary aim to eliminate the focus on quality after the fact or after the development in the Agile world. So dont try to define the word “Testing” in Agile. Try to define the word “Product Quality” in Agile. Your thought process will be better and you will get a real good sense of what you are doing. If we firefight against the Activity called Testing” this will mean to everyone that its a separate activity that have to be focused after some activity like development. I hope you all agree with me. Comments are welcome!!
I think we should concentrate on product quality for sure which is why we talk a lot about testing early – testing our assumptions, understanding etc. I think we should expand our definition of testing rather than treating it like something that ‘just’ happens. Testing is an activity, much like coding is an activity. Both activities need to happen – how and when we do them is what changes I think.
Thanks Janet. I like your thought and I agree. I wish we could find a new word in the dictionary which sounds more like testing early with more feedback loops, testing more frequently and testing continuously and testing assumptions.
My personal personal wish is to replace the traditional word testing with more frequent and continuous validation in new Agile world. I dont know any new English word though. Any English natives can help would be really great here.
It turns out to be quite an extensive definition. I do think some tinkering will make the definition stronger. What I find interesting in our field is that the test activities seem to go broader and beyond development nowadays. More testing is done on live products . A/B testing, dog feeding, Canary releases and monitoring usage of the product are examples of these. Would we say these practices are also part of agile testing?
If so I think the definition as I understand it is not broad enough. The definition now states ‘testing practices that occur from inception to delivery’. I think it goes beyond delivery. Or is that part of inception again? Maybe leaving the time frame and simply state that the testing practices can be continuous is an idea?
It was definitely harder than I thought it would be when we first started looking at the question.
The practices you mention such as A/B testing are definitely part of trying to get early feedback to see if we are building the right thing, but could be part of any testing effort although I have never seen it done on a phased and gated project.
Now the testing in production (which is really monitoring) does go beyond delivery and was traditionally part of operations …if it was done at all. The whole DevOps movement has brought it back to the team level. Something we hadn’t considered.
One of the reasons we put the time frame in, was to show that testing activities start at the very beginning, so I hate to lose that, but continuous might be good enough. Thanks for bringing this to our attention.
Eddy, that is an excellent point. We make a huge mistake when we focus on delivery and, once something is out the door, forget about it and start working on the next thing. It’s critical to learn how a feature gets used in prod. Lately my team has been looking at analytics, customer feedback and support tickets to learn whether we focused testing in the right areas. We’ve started doing A/B testing, but I see that more as a product/design activity than a true testing one. Still, it’s all about getting information, which is what testing is about too. I’m going to think how we can work that into the definition, I do think it’s important.
What a timely post. I will be having a session on Tuesday to identify what our testing agreements are as a team. I am taking a whole team approach in the session and I’m looking forward to see the outcome.
I do not see transparency above although it is part of our values. Should this be called out?
I know that full transparency will happen when the team discusses what type of testing is required for that story or sprint ( automation , manual , …).
I am wondering what your thoughts are on capturing any testing metrics for visibility or to help the team to use for continuous improvements?
Visibility / transparency very important for sure, and it’s really hard to put in everything into a definition. If you have our books we talk about visibility of testing in Chapter 7 & 8 of More Agile Testing, and Chapter 5 in Agile Testing where we talk about visibility and a bit about metrics. I’ve been meaning to write a blog post on some other thoughts as well, but …
Mostly though, the more visible you make testing activities, the more the team understands what’s involved and then you can have the conversation about how people can contribute.
My feeling is the conversation is the most important activity to do. There is so many ways we can build a quality product. Everyone on the team instinctively do that. It’s human nature to improve and make good stuff. We are unaware of how much is done until we have this conversation. The other aspect is that there is so many ways to build quality in that we have to have a whole team approach (period)! It is too difficult to have only one person to do this.
One other note, I feel that we are sometimes trying to justify why we need a tester on a team or why testers are needed. It is the beautiful mindset that a tester has that we need. One that has high value for quality one who sees every aspect that is needed for quality, that asks the great questions and challenges the team and sees the big and small picture. That mindset predominantly exists in tester but is not limited to that position. So let’s not look at why we need testers on a team .. let’s look at who has that mindset to help the team. More often than not it will be a tester.
I love that, Joanne! Who has that mindset and perspective – that’s the question to answer.
You mentioned metrics to help the team continuously improve. I’m a huge fan of doing small experiments. Look at your biggest problem, think of a hypothesis of what might make that problem smaller. Set SMART goals so that you’ll know whether your experiment is helping or not. I see most teams failing to define what success would look like and coming up with ways to measure progress. I don’t like arbitrary metrics for the sake of metrics, but when the team comes up with SMART goals together, it works – at least, in my experience.
thank you Lisa and Janet and the folks who responded to this blog. All great help!!
The philosophy of Design Thinking is very compatible. Testing has stepped into this space with examples, models in the inception. Interdisciplinary is a key word – not just cross functional. Validation through all those phases. Complexity seems key, and quality plays into that space for sure. Noodling the ideas, not concrete.
Hi Nick, I don’t know a lot about design thinking, but it does sound useful for testing activities and for finding ways to build in quality. As for complexity, I think the Cynefin model helps with that a lot. We can focus our testing on the complicated stories/features, and contribute to learning about the complex ones.
I find it an interesting topic Lisa. Some many parallels and also some more ideas. I’m preparing a blog article on the combination and linkages and see if there is any resonance towards it. I can say somethings are already being done 🙂 Some things to amplify and so new ideas.
Thanks, Nick. Pls ping me on Twitter when you post that blog!
This started an internal dialogue among some colleagues. We batted some of our ideas around (h/t to Jeffery Dunn) and this is where I ended up.
We summarize Agile Testing as:
Proactively assuring quality throughout the development process. We do this by ruthless inspecting and adapting with the goals of preventing defects and improving team efficiency. Key elements include building a common understanding around what we build, before we start building. And yes, we use tests as a feedback loop to confirm and prove our understanding.
I’m happy to have sparked a conversation :-). We are working on refining our definition based on some of the conversations as well.
I do like your definition with exception of a couple of things from a personal perspective. I like the word effectiveness over efficiency. Also, I’m not fond of the word ‘assuring’. The one thing I see missing is something about whole team.. It is implied I suppose, but it could be taken a different way. Thanks for sharing your ideas.
I am Kazuhiro SUZUKI. I work as an embedded software test engineer.
I am writing this comment to ask for permission for translating this post about agile testing into Japanese.
If possible, I would like to share the translation on my blog, which deals with software quality and software testing.
I look forward to your response.
Kazuhirio, Yes, you may translate – and we’d like it if you’d put the link on here as well. You have given us an idea so we may put up a link for future translations of the definition.
Thank you for rapid reply and permission!
I’ve just posted the translation below.
http://kzsuzuki.hatenablog.com/entry/2017/07/06/234332
My translation may not be perfect, so I hope some readers will correct mistakes.
I prefer the Testing Manifesto to the definition – more succinct, clear, and easier to comprehend – especially when supported by the bullets below in the post.
Thanks Todd. I do like the agile testing manifesto as well, and I think of it as complementary.
Did you read our second attempt? http://agiletester.mystagingwebsite.com/ever-evolving-never-set-stone-definition-agile-testing/
Yes, I do like that also. I think there’s room for multiple ways to communicate what we mean by the term “agile testing”. Janet and I did it in our first book with our 10 principles for agile testers. But people kept asking for an actual “definition”, so this is our attempt. I’m sure we have more modifications coming!
Thanks for the read and comment on my strategy Lisa!
I also believe in giving the acceptance tests to the developers and usually write it on the story as part of the planning session with them, however this is usually on a very high level. I then go into a bit more detail right at the start of the sprint when there’s nothing to test yet and compile more detailed test cases. I give this to them in a short, readable list on a piece of paper as if it’s in something like TestDirector, I can be guaranteed they won’t bother looking at it. This guarantees that they have built it in or included it in a test, so that leaves me free to find the real bugs.
The problem I have and don’t understand how it can be agile is that all automation tools that I’ve worked with requires concrete code in order to automate. I used Fitnesse years ago but felt it slowed me down and took way too much time updating the tests every time the code changes, which happens quite frequently when you are giving early feedback. I’m writing a post on how I do an automation test strategy to expand on my way of doing things.
I have concerns around the couple of terms used in definition. Defect Prevention and Defect Detection.
Software Testing is basically a defect detection activity. I think, it was never a defect prevention activity. A defect prevention activity will be part of software development process, which if followed will reduce the probability of introducing defects and avoid errors.
It is similar to the difference in Quality Assurance and Quality Control Process. Defect Prevention is a Quality Assurance Process and Defect Detection (Testing) is a Quality Control Process.
These are some foundational definitions and should not be tweaked to accommodate the agile terminologies.
I believe Software Testing is performed on finished product to Detect Defects and only the when and how will change when performed as a part of agile process.
Sameer, We didn’t try to define software testing, although that is how some people take it. In one of my earlier comments, I suggested that we need to talk about the product rather than the software. Software these days (in many contexts), is a small part of product development. I hope you read all the comments because there are many different opinions on this definition and all are welcome.
I test. I test lots of things, and not only software. I test people’s assumptions every time I ask a question. I think if we limit our thinking to testing to only looking at the software after it has been delivered, we are missing great opportunities to make our product better.
A perfect definition on agile testing. Thank you for sharing.
The definition of “Agile Testing” in the ISTQB Glossary (https://glossary.istqb.org/en/search/) is: “Testing practice for a project using Agile software development methodologies, incorporating techniques and methods, such as extreme programming (XP), treating development as the customer of testing and emphasizing the test-first design paradigm. ” I think it’s a good idea to have nothing of methods and procedures in the definition, just to refer to them or to give a few examples.
I realize that ISTQB has their own definition. Our definition is based on our years of experience and those of community members who helped us shape it. Our next blog post https://agiletester.mystagingwebsite.com/ever-evolving-never-set-stone-definition-agile-testing/ has a bit more.
software testing company in India
Thanks for sharing such a nice information about Agile .
very interesting and informative.
please keep sharing.
The two images in this article don’t seem to have a text version (longdesc or alt). Unfortunately the first one is the main part of the article – the definition itself. This doesn’t cause me any problems but will affect other readers. Could this be added? Thanks
Thank you. There is alt text for both images, although not a long description.
Yes, but the Alt text doesn’t contain the same information as the image. At the moment the definition of agile testing it out of reach of screen reader users. Thanks
The evolved definition is accessible and is available here https://agiletester.mystagingwebsite.com/ever-evolving-never-set-stone-definition-agile-testing
Great share! Thanks for the information. Keep sharing!
Very Nice and well explained!!