Slaying performance dragons

Always speak politely to an enraged dragon.”
Steven Brust

Regular readers will have noticed my penchant for referring to the Pareto Principle (aka the 80/20 rule). I’m a big believer in focusing on streamlining the high-use or highly important items of OSS implementations. OSS are simply too big and too complex to get bogged down in the minutiae.

But for those with a slant for the details, I’m clearly missing a very, very important point for CSPs with a high volume of transactions. This blog from 37signals about slaying “Dragons on the far side of the histogram” will be music to the ears of detailed-minded readers.

So if 37signals gets hit with 50 million requests a day, it’s quite likely that OSS at large CSPs get orders of magnitude larger again.

To quote their blog, “But when you process about fifty million requests a day, there’s still an awful lot of requests hidden on the far side of the 99th. And there, young ones, is where the dragons lie.

A while back we started shining the light into that cave. And even while I expected there to be dragons, I was still shocked at just how large and plentiful they were at our scale. Just 0.4% of requests took 1-2 seconds to resolve, but that’s still a shockingly 200,000 requests when you’re doing those fifty million requests.

Yet it gets worse. Just 0.0025% of requests took 10-30 seconds, but that’s still a whooping 1,250 requests. While some of those come from API requests that users do not see directly, a fair slice is indeed from real, impatient human beings. That’s just embarrassing! And a far, far away land from that pretty picture painted by the 60ms mean. Ugh.

Finally, there was the true elite: The 0.0001%, for a total of 50 instances. Those guys sat and waited between 30 and 60 seconds on their merry request to complete. Triple ugh.

To translate that into unhappy customers, that’s a lot per day on 50 million transactions.

How does that map to your OSS performance tuning? How far past the 80th percentile, or even the 99th percentile, do you shine a light?

Yes, I admit it. You details-guys are an absolutely vital component of delivering high levels of satisfaction to a CSP‘s customer base. 😀

Read the Passionate About OSS Blog for more or Subscribe to the Passionate About OSS Blog by Email

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.