Social TV app Zeebox (recently rebranded as Beamly) thought it was doing the right thing by moving from a tabbed interface to a navigation drawer, but the results were catastrophic — customer engagement dropped by half! Zeebox scrambled to issue an update that restored the tabbed UI. A second attempt proved equally disastrous. What started out as an attempt to embrace a widely used design pattern revealed that such best practices are contextual, not universal.”
Tony Costa
, here on Forrester.

The story above is a really interesting one from an OSS context. It makes me wonder so many things:

  • Whether OSS vendors are able to measure customer engagement of every operator using their products across all customers
  • Whether customers would allow their usage patterns of the OSS products to be collated and any insights derived be shared for the greater good of all customers of those products
  • Whether customers would allow live feeds of usage data to go back to base (ie a link back to the vendor) if they knew it was helping to improve the product or their ability to use it efficiently
  • Whether “customer engagement” is the best indicator for OSS products or whether efficiency metrics are more relevant (since the customer doesn’t allow their operators to opt-out from using the OSS tools, as opposed to discretionary products like Beamly in the example above)
  • Whether OSS vendors conduct A/B testing with real operators to determine whether proposed user interfaces (UI) are beneficial or catastrophic
  • Whether OSS vendors conduct any form of usability analysis at all. For example, a common form would be user questionnaires. But since certain cultures are predicated on respectful responses or saving face if the answer is unknown or unflattering, would questionnaires provide accurate data
  • Vendors certainly wouldn’t be polling the operators hourly, daily, weekly, so would the cycle time be fast enough, especially upon cutover to a new style of UI
  • Monitoring usage analytics to identify catastrophic usability degradation after cutover to a new UI is one thing, but do vendors have the contingency plans so that they’re not scrambling to release a roll-back or produce an alternate UI if a catastrophe occurs
  • Do vendors hold focus groups to gauge whether the UI is likely to be better, or give them a chance to refine it, long before reaching the cutover stage
  • Are focus groups going to produce reliable results in the design and prototype phase

I can’t possibly talk for all vendors/products but I have seen some OSS UIs that are catastrophic (IMHO) but the vendors remain blissfully unaware.

Read the Passionate About OSS Blog for more or Subscribe to the Passionate About OSS Blog by Email

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.