Vendor selection reference checks

Weaknesses seem to stick in my mind … I’d have to really think about any strengths
Actual reference check response

An past customer of mine recently asked for some guidance on what questions he (a representative of a CSP) should be asking his counterpart CSP that has already selected the OSS vendor that he’s evaluating.

I suggested that the following are the major classifications of questions you might like to ask, if you’re allowed to extend to the service wrapper built around the vendor’s OSS:

1. Planning
2. Change management
3. Gathering requirements
4. Business processes
5. Vendor references
6. Implementation structure and strategies
7. Operational structure and strategies
8. Testing structure and strategies
9. Data migration strategies
10. Training and knowledge transfer strategies
11. Future roadmap
12. Product fit-for-purpose

This set of questions is quite extensive (but I’m sure it’s still missing lots more), so out of courtesy to the other CSP, you might like to cull it quite a bit to only include the points that are most relevant to you.

1. Planning
• Can you broadly describe how you ran the network previously (eg via NMS or via an earlier OSS) and what your biggest objectives were when implementing your new OSS projects
• What were the most important things that you planned for
• Did you do an impact analysis to identify which parts of your business would be impacted
• Was there anything that you overlooked during the planning process
• How did you identify the best vendors to short-list (eg via RFI / EOI / RFP, doing your own research, etc)
• What were the most important things you learnt from doing this project
• Was it easy for you to develop a compelling business case to justify your project
• What were the features that were most relevant to the project sponsors before they approved the project
• Did you prepare an initial risk analysis and did you undertake ongoing risk management
2. Change management
• Did you follow a known change management process like Kotter
• How much time and effort did you need to invest in organisational change management
• How broad did you engage with stakeholders (eg all business units, some business units or only the project team and the people responsible for sponsoring the project)
• Did you have a dedicated change management group to help the project team communicate to your whole organisation
• How did you plan for training / knowledge-transfer to your staff from the vendor
• Did you plan your change management strategies before or during the project
3. Gathering requirements
• What process did you follow for gathering requirements
• Did you focus on business imperatives or technical / functional requirements
• Did you group your requirements around your organisation’s most important use cases
• Roughly how many stakeholders were involved in preparing your list of requirements
• In hindsight, are there any requirements that you now wish you had requested from the vendors
4. Business processes
• Did you already have strong business processes in place before your OSS project
• Did you follow a standard such as TMF’s eTOM?
• How deep did you go in your process planning (eg to Work Instruction level, or high-level)
• Did your business processes have to change much after commissioning your new OSS
• Roughly how much of the business process re-engineering (BPR) did you do and how much did the vendor do (eg did the vendor do roughly 60% of BPR and your experts had to contribute 40%)?
5. Vendor references
• What functionality / modules did the vendor implement for you? [Often the vendor has commissioned a completely different set of modules, which may be more advanced/mature than the modules that they are offering to the new CSP. For example, a particular vendor may specialise in alarm/fault management but also have rudimentary inventory management tools that they’re offering to the new CSP]
• What type of network do you operate (or more particularly what segment of the network is now managed by the vendor’s OSS). [Sometimes the vendor’s products are well suited to some network types (eg circuit-oriented) but not others (eg packet-switched). ]
• What type of services do you offer (or more particularly what sub-set of services are now managed by the vendor’s OSS)
• Can you reveal the financial cost of doing this project with the vendor? If there are restrictions on revealing financial details, what are the other dimensional data that can be divulged (eg number of users, number of interfacing systems, number of devices, number of transactions such as alarms, etc)?
• How much cost was direct (eg the amount in the contract with the vendor), how much was project variations and how much was internal cost (eg your operator resources had to do activities)
• How would you rate the vendor’s overall performance (on a scale from 1-10)?
• How would you rate the vendor’s quality of work (on a scale from 1-10)?
• How would you rate the vendor’s staff (on a scale from 1-10)?
• How would you rate the vendor’s ability to interact with your in-house staff, including their willingness to help guide you through your internal issues that arise on OSS projects (on a scale from 1-10)?
• How would you rate the vendor’s conflict resolution and change management skills (on a scale from 1-10)?
• How would you rate the vendor’s ability to manage the data migration process (on a scale from 1-10)?
• How would you rate the vendor’s ability to manage the solution cutover process (on a scale from 1-10)?
• How would you rate the vendor’s:
o Technical skills / knowledge
o Operational skills / knowledge
o Ability to understand your unique organisational features such as networks, processes, naming conventions, etc
o Ability to map your features into their tool-sets to produce a solution that was easy to understand and transition to by your operators
o Ability to map business needs into technical solution rather than forcing the business needs to retro-fit into the technical solution
• What are the strengths / weaknesses of the application suite. For example, was the product functional out of the box or did it require significant local configuration before it was able to do anything tangible, have you noticed major performance degradations, has it been intuitive for users to pick up, etc
• What are the strengths / weaknesses of the implementation (eg time / cost slippage, implementation / management methodologies, etc)
• What are the strengths / weaknesses of the ongoing support including ticket turnaround
• Ability to meet deadlines
• Availability of suitably skilled resources throughout the whole life-cycle (eg grads assigned to the project vs highly experienced experts)
• Did the supplied test cases provide suitable test coverage or have issues since been identified from areas that were testing blackspots
• Did the training provide your users with an acceptable level of knowledge given any time/budget constraints that you may’ve had?
• Was the training generic in nature or contextually relevant for your organisation
• How would you rate the level of ongoing support
• Was the vendor ambitious in seeking scope creep, logging change requests, etc versus accepting configuration changes / evolution within the scope of the initial contract
• Anything that you would do differently if implementing this project again
• How did you conduct the vendor selection process (eg EOI, RFP, direct vendor contact, etc). Did you have any learnings from the vendor selection process that you can recommend to us
• Have you collected any KPIs before / after the implementation of the OSS that show any improvements / degradations in performance (eg customers provisioned per day, fault rectification times, etc)
• Relating to the item above, when compared with your initial business case, would you suggest that the vendor’s solution has been recognised at executive level as being good value for investment, even if financial figures such as ROI is difficult to ascertain
• Is there anything you would like to add concerning this vendor or anything in particular that we should be aware of
• Having been through an OSS transformation, do you have any recommendations regarding organisational or operational structures or re-structuring that has arisen as a result of the introduction of the vendor’s tools (eg reallocating staff between business units, creation of a dedicated OSS team, resources to handle the support process, ongoing configuration or process refinements, implementation of ongoing projects, etc)
• Some vendors are adept at building in hooks that make their tools nearly impossible to remove from a CSP‘s environment once implemented. Some vendors also devise products that need cascading upgrades (ie if one module is upgraded then all other adjacent products also need upgrading), introducing cascading life-cycle costs.
6. Implementation team structure and strategies
• How did you organise your teams for this project? Did you have a dedicated project team as well as ongoing operations teams?
• What types of people did you need on your project team (eg project management, network experts, process experts, trainers, testers, IT administrators, etc)
• Was it difficult to get operational resources to assist the vendor to do the implementation
• Were you able to provide all of the expertise that the vendor requested
• Did you have experienced OSS implementation resources on your team or did you only have OSS operators on your team [Noting that there is a big difference in skill-sets required between building and operating an OSS]
• Looking back, did you or would you use independent expert OSS consultants to assist you with the OSS project to help fill in any gaps
• What areas would you use independent experts (eg testing, strategy, implementation, data migration, process mapping, risk management, etc)
7. Operational structure and strategies
• How did you structure the business units to handle your OSS (eg. Dedicated OSS team, network operations, IT, service order processing, field work-force, etc)
• Did you have to make much of a change in organisation structure to handle the OSS project
• Looking back, was there anything you would’ve done differently regarding your ongoing operations
8. Testing structure and strategies
• What testing structure did you follow
• Were you involved in all stages of testing or just User Acceptance Testing
• Did you design the UAT test cases or did the vendor do them for you
• Did you plan your testing around your most important use cases? If so, what were they
• Did you and the vendor use automated testing techniques to simplify regression testing during project implementation
• Were there many unresolved tests / bugs that remained open before go-live
• Was the vendor very responsive to resolving any bugs
• Did you start planning your testing late into the project or as a very early stage
9. Data migration strategies
• Did you or the vendor plan the data migration
• Did they require more time from your network experts than you planned
• Did you have to make any changes in your naming conventions to support the OSS
• Is there any advice you would give us regarding data migration strategy, skills or activities
10. Training and knowledge transfer strategies
• Was your team already experienced with using OSS tools before the start of the project
• Did you plan your training or did the vendor create the training plan
• What type of training did you undertake (eg coursework, train-the-trainer, on-the-job knowledge transfer by working side-by-side with the vendor)
• Is there any advice you would give us about training
11. Future roadmap
• Have you prepared a future roadmap
• Have you initiated that roadmap, or is it because of the vendor’s features not all being available yet
• What new technologies can you see that might impact the future OSS roadmap (eg network virtualisation techniques such as SDN/NFV, cloud services and orchestration, etc)
• Does this vendor have a clear vision for the future OSS that you are planning for
• What are the biggest risks you can imagine for your future OSS
12. Product fit-for-purpose
• There are so many here that are so specific to products and requirements that I’ll leave this one to you. I’m sure you’ll have a good feel for the important questions. Most specifically, you’re looking to get an understanding of what aspects are fit for purpose and what aren’t (if any)
13. Other
• Are there any final suggestions that you could give us (eg recommendations on what went well and what could’ve been done better during your OSS project)

I’m sure you can come up with lots of others. What are the big ones I’ve missed in your opinion?

If this article was helpful, subscribe to the Passionate About OSS Blog to get each new post sent directly to your inbox. 100% free of charge and free of spam.

Our Solutions


Most Recent Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.