How big is too big for OSS?

This sounds like a joke now – “there was a website that listed every single website that there was”. Yahoo actually worked pretty well when there were 20,000 websites, just as a book shop with 20,000 titles works pretty well. But the Yahoo directory peaked at 3.2m sites and at that that point it definitely didn’t work – you can’t possibly scroll past that many entries (though it lingered on in a half-life until Marissa Mayer shut it down this year). And in the meantime, Google invented PageRank, coming at the problem from an entirely different direction. Suddenly, search worked, and that seemed like the answer.
Today, app stores look a lot like the Yahoo of 20 years ago, and they don’t work for the same reasons – you can browse 20,000 apps but not a million. Hierarchical directories don’t scale. And so while it’s easy to make a list of things that Apple and Google should fix on their app stores, that misses the point – it’s like making a list of ways that the Yahoo home page should have been better. You might have been right but the answer was still Google. (I suspect that the same applies, just a little, to the current moves towards app search and deep linking, incidentally. PageRank uses the signal of links between pages – the ability to link of itself is only half the picture.) This is one reason mobile messaging apps are so hot – because they might become acquisition and discovery channels.
However, I think our preoccupation with the problems of apps and app stores and with the ways that they broke Google masks a deeper issue – that Google didn’t really solve the problem either. Or rather, it moved the problem. Google is very good at giving you what you’re looking for, but no good at all at telling you what you want to find, let alone things you didn’t know you wanted
.”
Benedict Evans
here.

Extending this concept to our industry, OSS actually works pretty well when there are up to 20,000 devices stored in a database of inventory items. But we’ve all worked on inventory management tools that have so many records that regardless of how much database optimisation you do, you still might as well head out for a coffee rather than sitting at your desk waiting for a query to return its results.

Okay, 20k devices doesn’t sound like it could overwhelm a database, but when you factor in multiple cards per device, with tens of ports per card, tens/hundreds of sub-interfaces or virtual ports per port/interface and you have extraordinarily large quantities of configuration items.

Also consider the amount of effort required to curate that data. Even if you can discover most of the data via a programmable interface, there are still additional fields that require manual data population (eg operational status [eg out of service, in testing, etc], contact details, maintenance, lifecycle / end-of-life, etc).

To my knowledge (and I’d be delighted to be informed otherwise), every inventory management tool takes the Yahoo approach of trying to maintain a hierarchical directory of inventory. When their customers (ie OSS operators) complain that queries are too slow, the standard answer is that they’re limited by the capabilities of the underpinning database. That answer is correct, but nobody appears to be doing a Google and coming at the problem from an entirely different direction.

The virtualised world of SDN/NFV promises to rapidly increase the volumes of configurable items beyond where we are today. It will also introduce far faster lifecycles (ie create and tear-down) so the inventory management problem we find ourselves in now will only magnify into the future (see more on the touchpoint explosion here).

I’ve long been an advocate of a different approach to inventory management, based on a variant of search technology rather than duplicated database of records (ie hierarchical directories). What Ben talks about in detail in his blog (here’s the link again if you want to read it in its entirety) is also relevant to us in OSS – “Google is very good at giving you what you’re looking for, but no good at all at telling you what you want to find, let alone things you didn’t know you wanted.” OSS vendors (inventory management vendors in particular) should definitely be monitoring the developments in this space because that’s exactly where we need to get to as well.

Read the Passionate About OSS Blog for more or Subscribe to the Passionate About OSS Blog by Email

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.