Granted, these quotes are made up. But the idea behind them is not and sometimes seen in testing projects. The way I see it, it boils down to the topic of test oracles. To quote Wikipedia: a test oracle “is a mechanism for determining whether a test has passed or failed.” Back in my university days Wikipedia was not really a proper source to quote from, but times have changed and I think that we are having a good working definition here. In addition, this post is not really about defining what a testing oracle actually is (I simply assume that you are familiar with the concept), but what different test oracles are out there. The reason for this post is that I see a lot of situations where people assume that there is actually just one test oracle available: the almighty requirement. While the requirement or user story certainly is a test oracle, it is no way the only one that should be taken into consideration. Especially not in agile environments where you might have something like the 3C-principle in place: card, conversation and confirmation. Or in a nutshell: You are nuts (pun intended) if you write everything down that you talked about and agreed upon. To give you an example: if you are working on a web application, it is pretty likely that the screen resolution is not mentioned in every requirement. It might be part of some other technical document, it might be just something the PO said or it might be just something that you know. Regardless of the source, you should take it into consideration. Those three sources are good examples of the three categories of test oracles that are out there: Documents, people and knowledge.
This is probably the most obvious category of test oracles. There can be a variety of documents that can be used as a test oracle and that should be considered when designing your tests. Documents that come immediately to mind are:
- Technical documentation
- Style guides
- Testing notes from earlier test executions
Some of these might be pretty stable, standards or technical documentation for example. Others, like requirements or testing notes are more short-lived and more prone to change. On a side note: In this case it is pretty useful if you have a tool like TestBench that offers traceability, so you can easily see where a tests originates from and what other requirements might be helpful as a testing oracle, just to make sure that nothing was changed that was designed a certain way for purpose. But the test cases linked to the requirements also can and should be used to substantiate the interpretation of the requirements and also helps to implement or support Test Driven Development on requirements level.
This category is more of an indirect nature and using it as an information source might lead to further documents, but you want to keep things lean, don’t you? Before starting to test, there are people that you should consult and use their answers as a test oracle:
- Product owners
Not all of these are easily accessible and you might get contradictory information, but hey, that is actually a good sign. These things should be clarified before developing and testing and not after everything went live. What all of these people have in common, though, is that they are experts for their topic.
Speaking of experts: Sometimes we as testers tend to not see ourselves as experts (hello impostor syndrome), but often we are in line with the other experts, so it is perfectly alright to interview ourselves instead of others, also known as taking the time to think about the testing challenge ahead from different angles. If that knowledge stems from experience with your software or from having read something some time ago is not to be neglected, though. Reading is fine, but having first-hand experience is usually the better source of information.
So which test oracle should you use? The answer is as always: It really depends on your context. Especially knowledge is probably always involved as it is mainly of an implicit nature, while the others are more explicit. In a way test oracles are like every other piece of information out there, you should validate it and see if it can be trusted and used. A style guide written in 2004 is probably not as reliable as yesterday’s answer of the product owner, even if it a written and not an oral source.
The lists provided above are by no means complete and you can probably name a few more valid for your context. Regardless of being listed here or not, no test oracle should be used in its own since customers do not really care if it was written in a certain document or not. And that’s why we as testers should take several test oracles into account, depending on the focus of our testing.
Christian Kram for TestBench Cloud Services