Backtracking

“Social TV app Zeebox (recently rebranded as Beamly) thought it was doing the right thing by moving from a tabbed interface to a navigation drawer, but the results were catastrophic — customer engagement dropped by half! Zeebox scrambled to issue an update that restored the tabbed UI. A second attempt proved equally disastrous. What started out as an attempt to embrace a widely used design pattern revealed that such best practices are contextual, not universal.”
Tony Costa, here on Forrester.

The story above is a really interesting one from an OSS context. It makes me wonder so many things:

• Whether OSS vendors are able to measure customer engagement of every operator using their products across all customers
• Whether customers would allow their usage patterns of the OSS products to be collated and any insights derived be shared for the greater good of all customers of those products
• Whether customers would allow live feeds of usage data to go back to base (ie a link back to the vendor) if they knew it was helping to improve the product or their ability to use it efficiently
• Whether “customer engagement” is the best indicator for OSS products or whether efficiency metrics are more relevant (since the customer doesn’t allow their operators to opt-out from using the OSS tools, as opposed to discretionary products like Beamly in the example above)
• Whether OSS vendors conduct A/B testing with real operators to determine whether proposed user interfaces (UI) are beneficial or catastrophic
• Whether OSS vendors conduct any form of usability analysis at all. For example, a common form would be user questionnaires. But since certain cultures are predicated on respectful responses or saving face if the answer is unknown or unflattering, would questionnaires provide accurate data
• Vendors certainly wouldn’t be polling the operators hourly, daily, weekly, so would the cycle time be fast enough, especially upon cutover to a new style of UI
• Monitoring usage analytics to identify catastrophic usability degradation after cutover to a new UI is one thing, but do vendors have the contingency plans so that they’re not scrambling to release a roll-back or produce an alternate UI if a catastrophe occurs
• Do vendors hold focus groups to gauge whether the UI is likely to be better, or give them a chance to refine it, long before reaching the cutover stage
• Are focus groups going to produce reliable results in the design and prototype phase

I can’t possibly talk for all vendors/products but I have seen some OSS UIs that are catastrophic (IMHO) but the vendors remain blissfully unaware.