“Testers! Break that software (as you must) and drive it to the ultimate — but don’t enjoy the programmer’s pain.”
Boris Beizer.
Are you new to the world of OSS and BSS testing and aren’t quite sure what testing is required? The following is a set of commonly conducted tests. Some may be relevant to your OSS, others may not:
- Factory Acceptance Testing (FAT) – This consists of stand-alone functionality testing that the COTS vendor will conduct to ensure the application/s are of suitably high quality prior to release to customers. If you’re using Commercially-available Off The Shelf (COTS) software, this phase of testing won’t be so relevant to you.
- Data Migration Testing (DMT) – In many cases, the process you undertake for data creation, migration, cleanse and load will need to be tested to ensure the loaded data allows the application/s to function as expected. This phase is normally conducted by the integrator.
- On-site product / system testing / Site Acceptance Test (SAT) – This is a form of stand-alone testing on modules or solution components of your OSS. This will allow you to verify that the basic installation has succeeded and each component is operating as expected (with data that is contextually relevant to your solution) before integrating with other components. This phase is also normally conducted by the integrator.
- System Integration Testing (SIT) – This is usually the most comprehensive stage of testing where the supplier and integrator test their end-to-end solution to ensure that it is functioning as expected across the whole system specification. The customer may opt to observe system integration testing, in part or in full. This stage also often evaluates the functional requirements from the business requirement definition. This is also sometimes known as Shake-out Testing when multiple vendor solutions are involved, possibly including network equipment and/or EMS (Element Management Systems).
- Redundancy and Failover Testing – If you have highly available (HA) mechanisms in your OSS environment, it is reasonable to expect that you and / or your integrator will perform failover testing to ensure that any failures on the primary instance of your OSS will allow business continuity on secondary instances. Note that this may include intra-site testing (if there are HA mechanisms within sites in case of a component failure) and / or inter-site testing (if there are HA mechanisms between sites in case of a data centre or cluster / node failure).
- Performance and Volume Testing (PVT) or Stress and Volume Testing (SVT) – This phase is designed to place your OSS under enough stress or load to replicate / exceed expected situations once your OSS goes live. This phase may allow the integrator to refine configurations to reduce any unexpected performance degradations. One of the key elements to consider in this phase is how you will synthesise the load. Examples might include specialist load generators, custom scripts that flood the OSS with data, etc. This phase is usually conducted by the integrator, although multi-vendor cooperation may be required sometimes.
- Release, Install and Back-out Testing – This phase may be required if you are performing a system upgrade to parts of your OSS. Install and back-out testing would be done on a non-production instance of your OSS before attempting the upgrade to your live OSS. It’s common for new application releases / patches to be tested in a non-production environment before performing the upgrade on production systems
- Security Testing – Security and penetration testing is an important phase of your OSS testing. Once commissioned, your OSS could become an incredibly powerful tool in the wrong hands, so vulnerability checks are vital. Whilst you may choose to ask your integrator to test and certify the security of the solution they’re commissioning for you, it is recommended to arrange a specialist security group to perform this testing phase for you. This group could either be your dedicated in-house security team (if you’re lucky) or via a specialist third-party security analysis firm.
- Non-Functional Testing (NFT) – This might represent a number of the test phases described above (eg SVT / PVT, security, etc)
- User Acceptance Testing (UAT) – The aim of UAT is to ensure that the solution delivers upon the functional and non-functional requirements that were agreed upon in the requirement definition. I have seen some customers delegate responsibility for preparing their UAT cases to the integrator, thus allowing the integrator to run a pre-UAT to ensure expected performance exists before handing over to the customer. However, integrator-provided UAT test cases may not necessarily check the boundary cases or use data that is contextually relevant to your use of the OSS. It is recommended to use experienced OSS or software testers who are familiar with the solution and ask them to work closely with the integrator as you develop and run your UAT test cases.
If you’re also interested in what types of PROD (Production) and non-production environments are commonly used to support OSS/BSS solutions, as well as the various associated test phases above, click here on “What OSS environments do you need?”.