“Beware of the problem of testing too many hypotheses; the more you torture the data, the more likely they are to confess, but confessions obtained under duress may not be admissible in the court of scientific opinion.”
Stephen M. Stigler.
In a recent post, entitled “Data Architecture Layers,” we discussed the need to architect data from the perspective of both format and context.
The context perspective is possibly even more important in testing.
When building up test scenarios there are many considerations to undertake, but when it comes to the data sets, context is hugely important. Firstly, we need to consider the inputs/actions that drive the application through a set of operations/transactions/flows, but invariably there is also a need for well designed data sets that underpin these operations/transactions/flows.
Let’s say the test scenario involves creating a new service order through a network. Before you can conduct this test, you first have to build/simulate/emulate the network upon which the service is carried. If your underpinning network data doesn’t reflect the real environment, possibly because you’re not a subject matter expert (SME) on that particular type of network, then your test scenario is immediately compromised.
Where the context of the data becomes even more important is when you’re simulating situations in the network that test your exception handling and/or boundary cases. You often aren’t able to just take a live feed of existing data (eg an export from a production database instance) because it may not have the scenario that you’re looking to validate. You may have to craft specific data sets to test the application’s logic.
I’ve seen numerous cases where testing has failed to identify major logic flaws, not because the test scenario was invalid, but because the underpinning data was too unsophisticated. This is where developers and testers often need the assistance of SMEs to help build their scenarios.
To extend this concept further, the SME may even need to be an expert on the particular CSP’s environment. For example, naming conventions tend to be unique to each CSP. I’ve seen examples where tests have passed based when using generic naming in the test data, but have failed catastrophically when using a customer’s real naming conventions.