What will get your CEO fired?

Not sure whether you have a clear answer to that question – either the thought is enticing (you want the CEO to get fired), unthinkable (you don’t want the CEO fired) or somewhere in between.  You’d invariably get different answers from different employees within most organisations (you can’t please all of the people all of the time).

Today’s post also has no singular situation, so it poses a number of hypothetical questions. But let’s follow the thread of the question from a purely technical perspective (ie discounting inappropriate personal decisions, incompetence and the like).

If we look at the technical factors that could get the big boss fired, they’re probably only limited to:

  1. Repeated and/or catastrophic failure (of network, systems, etc)
  2. Inability to serve the market (eg offerings, capacity, etc)
  3. Inability to operate network assets profitably

Let’s start with catastrophic failures, ones where entire networks and/or product lines go down for long periods (as opposed to branches of networks becoming faulty or intermittently failing). We’ve had quite a few catastrophic failures in my region in the last few years.

Interestingly, we’re designing networks and systems that should be more reliable day-by-day. In all likelihood they probably are. But with increased system complexity and programmed automation, I wonder whether we’re actually increasing the likelihood of catastrophic failures. That is, examples of cascading failures that are too complex for operators to contain.

Anecdotal evidence surrounding the failures mentioned above in my area suggest that a skills-gap after many retirements / redundancies has been partly to blame. The replacements haven’t been as well equipped as their predecessors to prevent the runaway train of catastrophic failure.

Do you think a similar risk awaits us with the current trend towards build-it-yourself OSS? We have all these custom-built, one-of-a-kind, complex systems being built currently. What happens in a few years time when their designers / implementers leave? Yes, their replacements might be able to interrogate code repositories, but will they know what changes to rapidly make when patterns of degradation start to appear? Will they have any chance of inherently understanding the logic behind these systems like those who created them? Will they be able to stop the runaway train/s?

This earlier post discussed the build vs buy cycle the market goes through.

OSS build vs buy cycle

There are many pros and cons of the build vs buy argument, but at least there’s a level of repeatability and modularity with off the shelf solutions.

I wonder if the CEOs of today/tomorrow are planning for sophisticated up-skilling of replacements in future when they decide to build in-house today? Or will they just hand the replacements the keys and wish them luck? The latter approach keeps costs down, but it just might get the CEO fired!

We’ll take a closer look at the other two factors (“serve the market” and “profitability”) in the next few days.

 

 

 

If this article was helpful, subscribe to the Passionate About OSS Blog to get each new post sent directly to your inbox. 100% free of charge and free of spam.

Our Solutions

Share:

Most Recent Articles

No telco wants to buy an OSS/BSS

When you’re a senior exec in a telco and you’ve been made responsible for allocating resources, it’s unlikely that you ever think, “gee, we really

2 Responses

  1. The popular development methods of the day must surely shoulder some of the blame. Rather, those who allow their unfettered use may be in the spotlight.
    Top of my list is “Agile” development. It gets systems that work (hang together) out the door quickly. It actively favours working software over documentation. There is no need to have undocumented systems, but the bias is all too tempting; as soon as it “works”, declare it “on-budget” and get on with something new. What’s left in the heads of developers and users – as you say – will fade away and people will turn over. For those who are going red-in-the-face about now, I’m not bagging Agile, just those who allow it to destroy common sense and regard for the future. The temptation is great.
    Following closely behind is the uncertainty which can flow from micro-services based, decoupled solutions which are now commonplace. The benefits are many, including modular testing and scalability. Risk accrues from the exponential growth of the possible interactions (a bit like the network itself) and is partnered in crime often by the use of many 3rd party open-source services which may conceal tricky little behaviours which emerge under stress. Toss into the mix a few “improvements” to the services which introduce hard-to-find” side effects. At some time, under load, exponential runaway or deadlocks may occur. Sure, we can test, test and test again, but how far can you go; the budget’s burning down? We can be left in the same space – it’s working, so on with the next thing. Once more, not just a technical issue, but more one of inclination, investment and common sense in the face of urgency to have a solution. And once more vulnerable to knowledge-fade and turnover.
    How can our hapless CEO spot these traps? His generals may be delivering on time and budget, but are the short cuts rampant?

  2. Hi Steve,
    Brilliant stuff!!
    I’m completely with you on this one, as per this post https://passionateaboutoss.com/the-oss-tech-debt-wreck/
    And a related quote from Warren Buffett sums it up nicely, https://passionateaboutoss.com/the-chains-of-integration-are-too-light-until/

    Having said that, I’m actually quite a fan of agile… IF…….
    1) Those steering the product (or entire solution) are creating epics with a long-term view of where it is headed (not just meandering incremental tweaks)
    2) There is a culture of subtraction projects and re-factoring being as important as feature releases (https://passionateaboutoss.com/subtraction-projects/ )

    PS. I love your reference to open-source, where it’s tempting for devs to use as-is, but not understand the open-source code (or its peccadilloes) until the runaway train starts gaining momentum.

    Thanks for sharing Steve!!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.