“Faith is the great cop-out, the great excuse to evade the need to think and evaluate evidence.”
Evaluating OSS vendors is never an easy task. There are virtually no apples-for-apples comparisons, even within domains (eg alarm/fault management). Each product set has different features, support, etc.
Despite this, I have used an approach similar to this Vendor Selection Workflow for a number of customers
The key to this approach is to issue an Expression of Interest (EOI) pack to vendors to quickly identify the vendors that supply functions/benefits that are most closely aligned to the customer’s needs. [Note: On past projects, I’ve found that this method has identified 1-3 vendors of obvious closest fit out of an initial list of 10-20]. These vendors are short-listed and can be engaged more thoroughly on a proposal and product demonstrations. Once a preferred vendor is selected, the negotiations commence.
The Vendors / Products Matrix can also be used to help short-list the vendor / integrator that best fits your requirements.