What this blog post is really about is different approaches to documentation in testing. When dealing with testing in an agile work environment there is one thing I have seen people and projects forget about from time to time. The agile manifesto tells us that we should value “individuals and interactions over processes and tools”. Sometimes people get so much into tools they are convinced of that they let their tools dictate their processes and by that their interactions. So what we really need from a tool is that it doesn’t guide all of our actions, i.e. testing in our case, but support our testing. So instead of putting a lot of time into organizing and administrating our tools, they should support us with testing by enabling us to manage things like test data or information about whatever is necessary in your specific test environment.
In my experience there are two major fractions when talking about organizing your tests. On the one hand you have those promoting exploratory testing and those that are firm believers in scripted testing. To me that’s not really an “either or” decision but more of what suits me better in a given context right now. In exploratory testing things like test focus, test ideas and especially those notes you are taking while testing are things you might want to document. In scripted testing you might have more detailed ideas about preconditions, the exact steps you want to do and what exactly you are expecting to see from that isolated step you have.
So let’s see what information might be useful in those two contexts.
As we all know exploratory testing is not ad-hoc testing, so we need some kind of structure or test mission. This is something I like to have in a management tool. First of all it is way more legible than my handwriting, but more important it can be searched more easily. Something else that I have grown to value over time is traceability. When I see a test that I have done some time ago that is relevant to what I am testing right now, looking at what old user stories might be affected is helpful in evaluation risks and looking at other tests that might possibly be impacted.
So what I really want to take down is my testing mission: What will be the focus of my testing? What’s my time box for this test? What am I looking for? What will I be putting aside on purpose? I like mind mapping here as well, so I need some place to manage my mind maps, so why not as attachments in my testing tool?
Those are the things that I want to have ready before I start testing. But of course there are things that I want to take down while testing. Note-taking is an art of its own and the range of note-taking goes from pen and paper to screen capturing to using my testing tool directly. I like mixing up between those three. Pen and paper is more for things I can learn from and improve personally or generating new ideas, while screen capturing can be a nice addition for reproducing bugs. Using the tool is especially useful when noting down what has worked well or not so well. So this is already hinting at putting down information after testing. I am a huge proponent of debriefings after testing dealing with questions like “what did I do?”, “What struck me?” or “Where would I investigate further?". These are information that are useful and valuable to other team members as well, so they should be available to them as well.
Now scripted testing works a little bit different, but might need tool assistance even more. In scripted testing I want to predefine as much as possible before I start. Some people even go as far as saying that you need to write down so much that everyone can execute the test case. The focus of documentation shifts upfront when compared to exploratory testing.
So what information do we need to convey easily? Chances are high that you have some kind of predefined test environment so I want to choose between relevant options easily. So having those available instead of re-defining them every time is a must for a tool. Instead of providing ideas and focus, I need defined steps and a defined result. What really helps you safe time when writing tests cases is when you can reuse those steps? Those 4 steps for logging in will probably the same time and you don’t want to copy and paste those, will you? That would be a maintenance hell if you should ever change anything.
During test execution there will be less documentation, except for saying what steps you already tested and making a binary decision that it either worked or not.
Regardless of the process you are working with, may it be exploratory testing with less early-on documentation and more documentation during execution and afterwards or scripting testing with heavy up-front documentation and less later on, what you really need from your tool is adaptability. You want to configure the tool in a way that it helps you within your testing process and you don’t want it to define the process for you.
Christian Kram for TestBench Cloud Services