How and why we need to Challenge our OSS Beliefs

The OSS / BSS industry tends to be quite technical in nature. The networks we manage are technical. The IT stacks we build on are technical. The workflows we design are technical. The product offerings we coordinate are technical. As such, the innovations we tend to propose are quite technical too. In some cases, too technical, but I digress.

You would’ve no doubt noticed that the organisations garnering most attention in the last few years (eg the hyperscalers, Airbnb, Uber, Twitter, Afterpay, Apple, etc) have also leveraged technology. However, whilst they’ve leveraged technology, you could argue that it’s actually not “technical innovation” that has radically changed the markets they service. In most cases, it’s actually the business model and design innovations, which have simply been facilitated by technology. Even Apple is arguably more of a design innovator than a technology innovator (eg there were MP3 players before the iPod came along and revolutionised the personal music player market).

Projects like Skype, open-source software and hyperscaled services have made fundamental changes to the telco industry. But what about OSS/BSS and the telcos themselves? Have we collectively innovated beyond the technical realm? Has there been any innovation from within that has re-framed the entire market?

As mentioned in an earlier post, I’ve recently been co-opted onto the TM Forum’s Transformation User Group to prepare a transformation framework and transformation guides for our industry. We’re creating a recipe book to help others to manage their OSS/BSS/telco/digital transformation projects. However, it only dawned on me over the weekend that I’d overlooked a really, really important consideration of any transformation – reframing!

Innovation needs to be a key part of any transformation, firstly because we need to consider how we’ll do things better. However, we also need to consider the future environment in which our transformed solutions will operate within. Our transformed solution will (hopefully) still be relevant and remain operational in 5, 10, 15 years from now. For it to be relevant that far into the future, it must be able to flexibly cater for the environmental situation in the future. Oh, and by the way, when I say “environment,” I’m not talking climate change, but other situational change. This might include factors like:

  • Networks under management
  • Delivery mechanisms
  • Business models
  • Information Technology platforms 
  • Architectural models
  • Process models
  • Staffing models (vs automations)
  • Geographical models (eg local vs global services)
  • Product offerings (driven by customer needs)
  • Design / Develop / Test / Release pipelines
  • Availability of funding for projects, and is it capex, opex, clip-the-ticket, etc
  • Risk models and risk adversity level
  • etc

Before embarking on each transformation project, we first need to challenge our current beliefs and assumptions to make sure they’re still valid now, but can remain valid into the next few years. Our current beliefs and assumptions are based on past experiences and may not be applicable for the to-be environment that our transformations will exist within.

So, how do we go about challenging our own beliefs and assumptions in this environment of massive, ongoing change?

Well, you may wish to ask “re-framing questions” to test your beliefs. These may be questions such as (but not limited to):

  1. Who are our customers (could be internal or external) and how do they perceive our products? Have we asked them recently? Have you spent much time with them?
  2. What is our supply chain and could it be improved? Are there any major steps or painful elements in the supply chain that could be modified or removed
  3. Are these products even needed under future scenarios / environments
  4. What does / doesn‘t move the needle? What even is “the needle”
  5. What functionality is visible vs invisible to customers 
  6. What data would be useful but is currently unattainable 
  7. Do we know how cash flows in our clients and our own companies in relation to these solutions? Specifically, how value is generated
  8. How easy are our products to use? How long does it take a typical user (internal / external) to reach proficiency 
  9. What personas DO we serve? What personas COULD we serve? What new personas WILL we serve
  10. What is the value we add that justifies our solution’s existence? Could that value be monetised in other ways
  11. Would alternative tech make a difference (voice recognition, AI, robotics, observability, biometric sensors, AR/VR, etc)
  12. Are there any strategic relationships that could transform this solution
  13. What does our team do better than anyone else (and what are they not as strong at)
  14. What know-how does the team possess that others don’t
  15. What features do we have that we just won’t need? Or that we absolutely MUST have
  16. Are there likely to be changes to the networks we manage
  17. Are there likely to be changes to the way we build and interconnect systems
  18. Where does customer service fall on the continuum between self-service and high-contact relationships
  19. What pricing model best facilitates optimal use of this solution (if applicable) (eg does a consumption-based usage / pricing model fit better than a capital investment model?). As Jeff Bezos says, “Your margin is my opportunity.” The incumbents have large costs that they need to feed (eg salaries, infrastructure, etc), whereas start-ups or volume-models allow for much smaller margins
  20. Where are the biggest risks of this transformation? Can they be eliminated, mitigated or transferred
  21. What aspects of the solution can be fixed and what must remain flexible
  22. What’s absorbing the most resources (time, money, people, etc) and could any of those resource-consumers be minimised, removed or managed differently

It also dawns on me when writing this list that we can apply these reframing questions not just to our transformation projects, but to ourselves – for our own personal, ongoing transformation.

I’d love to get your perspective below. What other re-framing questions should we ask? What re-framing exercise has fundamentally changed the way you think or approach OSS/BSS/personal transformation projects?

How to Approach OSS Vendor Selection Differently than Most

Selecting a new OSS / BSS product, vendor or integrator for your transformation project can be an arduous assignment.

Every network operator and every project has a unique set of needs. Counter to that, there are literally hundreds of vendors creating an even larger number of products to service those widely varied sets of needs.

If you’re a typical buyer, how many of those products are you already familiar with? Five? Ten? Fifty? How do you know whether the best-fit product or supplier is within the list you already know? Perhaps the best-fit is actually amongst the hundreds of other products and suppliers you’re not familiar with yet. How much time do you have to research each one and distill down to a short-list of possible candidates to service your specific needs? Where do you start? Lots of web searches?

Then how do you go about doing a deeper analysis to find the one that’s best fit for you out of those known products? The typical approach might follow a journey similar to the following:

The first step alone can take days, if not weeks, but also chews up valuable resources because many key stakeholders will be engaged in the requirement gathering process. The other downside of the requirements-first approach is that it becomes a wish-list that doesn’t always specify level of importance (eg “nice to have” versus “absolutely mandatory”).

Then, there’s no guarantee that any vendor will support every single one of the mandatory requirements. There’s always a level of compromise and haggling between stakeholders.

Next comes the RFP process, which can be even more arduous.

There has to be an easier way!

We think there is.

Our approach starts with the entire 400+ vendors in our OSS/BSS Vendor Directory. Then we apply two rounds of filters:

  1. Long-list Filter – Query by high-level product capability as per diagram below. For example, if you want outside plant management, then we filter by 9b, which returns a list of over 60 candidate vendors alone, but we can narrow it down further by applying filters 10 and 14 as well if you need these functionalities
  2. Short-list Filter – We then sit with your team to prepare a list of approx. 20 high-level questions (eg regions the vendor works in, what support levels they provide, high level functional questions, etc). We send this short questionnaire to the long-list of vendors for their clarification. Their collated responses usually then yields a short-list of 3-10 best-fit candidates that you/we can then perform a deeper evaluation on (how deep you dive depends on how thorough you want the review to be, which could include RFPs, PoCs and other steps).

The 2-step filter approach is arguably even quicker to prepare and more likely to identify the best-fit short-list solutions because it starts by assessing 400+ vendors, not just the small number that most clients are aware of.

The next step (step 5 in the diagram above) also uses a contrarian approach. Rather than evaluating via an RFP that centres around a set of requirements, we instead identify:

  • The personas (people or groups) that need to engage with the OSS/BSS
  • The highest-priority activities each persona needs to perform with the OSS/BSS
  • End-to-end workflows that blend these activities into a prioritised list of demonstration scenarios

These steps quickly prioritise what’s most important for the to-be solution to perform. We describe the demonstration scenarios to the short-listed vendors and ask them to demonstrate how their solutions solve for those scenarios (as best they can). The benefit of this approach is that the client can review each vendor demonstration through their own context (ie the E2E workflows / scenarios they’ve helped to design).

This approach does not provide an itemised list of requirement compliance like the typical approach. However, we’d argue that even the requirement-led approach will (almost?) never identify a product of perfect fit, even if it’s filled with “Will Comply” responses for functionality that requires specific customisation.

Our “filtering” approach will uncover the solution of closest fit (in out-of-the-box state) in a much more efficient way.

We should also highlight that the two chevron diagrams above are just sample vendor-selection flows. We actually customise them to each client’s specific requirements. For example, some clients require much more thorough analysis than others. Others have mandatory RFP and/or business case templates that need to be followed.

If you need help, either with:

  • Preparing a short-list of vendors for further evaluation down from our known list of 400+; or
  • Need help to perform a more thorough analysis and identify the best-fit solution/s

then we’d be delighted to assist. Please leave us a note in the contact form below.

New. Just Launched – Mastering Your OSS (the eBook version)

Featured

We’re excited to announce that we’ve just launched the eBook version of Mastering Your OSS.

Click on the link above or here to read all about it.

Time to Kill the Typical OSS Partnership Model?

A couple of years ago Mark Newman and the content team at TM Forum created a seminal article, “Time to Kill the RFP? Reinventing IT Procurement for the 2020s.” There were so many great ideas within the article. We shared a number of concordant as well as divergent ideas (see references #1, #2, #3, #4, #5, #6, and others).

As Mark’s article described, the traditional OSS/BSS vendor selection process is deeply flawed for both buyer and seller. They’re time-consuming and costly. But worst of all, they tend to set the project on a trajectory towards conflict and disillusionment. That’s the worst possible start for a relationship that will ideally last for a decade or more (OSS and BSS projects are “sticky” because they’re difficult to transform / replace once in-situ).

Partnership is the key word in this discussion – as reiterated in Mark’s report and our articles back then as well.

Clearly this message of long-held partnerships is starting to resonate, as we see via the close and extensive partnerships that some of the big service providers have formed with third-parties for integration and other services. 

That’s great…. but…… in many cases it introduces its own problem for the implementation of OSS and BSS projects. A situation that is also deeply flawed.

Many partnerships are based around a time and materials (T&M) model. In other words, the carrier pays the third-party a set amount per day for the number of days each third-party-provided resource works. A third-party supplies solution architects at ($x per day), business analysts at ($y per day), developers at ($z per day), project managers at… you get the picture. That sounds simple for all parties to wrap their head around and come to mutually agreeable terms on. It’s so simple to comprehend that most carriers almost default to asking external contractors for their daily charge-out rates.

This approach is deeply flawed – ethically conflicted even. You may ask why…. Well, Alan Weiss articulates it best as follows:

When you charge by the hour, you’re in ethical conflict with the client. You only receive larger pay for the longer you’re there, but the client is better served by how quickly you can meet objectives and leave.

Complex IT projects like OSS and BSS projects are the perfect example of this. If your partners are paid on a day rate, they’re financially incentivised for delays, bureaucracy, endless meetings and general inefficiency to prosper. In big organisations, these things tend to already thrive without any incentivisation!

Assuming a given project continues at a steady-state of resources, if a project goes twice as long as projected, then it also goes 100% over the client’s budget. By direct contrast, the third-party doubles their revenue on the project.

T&M partnership models disincentivise efficiency, yet efficiency is one of the primary reasons for the existence of OSS and BSS. They also disincentivise reusability. Why would a day-rater spend the extra time (in their own time) to systematise what they’ve learnt on a project when they know they will be paid by the hour to re-invent that same wheel on the next project?

Can you see why PAOSS only provides scope of work proposals (ie defined outcomes / deliverables / timelines and, most importantly, defined value) rather than day-rates (other than in exceptional circumstances)??

Let me cite just one example to illustrate the point (albeit a severe example of the point).

I was once assisting an OEM vendor to implement an OSS at a tier-1 carrier. This vendor also provided ongoing professional services support for tasks such as customisation. However, the vendor’s day-rates were slightly higher than the carrier was paying for similar roles (eg architects, developers, etc). The carrier invited a third-party to perform much of the customisation work because their day-rates were lower than the OEM.

Later on, I was tasked with reviewing a customisation written by the third-party because it wasn’t functioning as expected. On closer inspection, it had layers of nested function calls and lookups to custom tables in the OEM’s database (containing fixed values). It comprised around 1,500 lines of code. It must’ve taken weeks of effort to write, test and release into production via the change process that was in place. The sheer entanglement of the code took me hours to decipher. Once I finally grasped why it was failing and then interpreted the intent of what it should do, I took it back to a developer at the OEM. His response?

Oh, you’ve gotta be F#$%ing kidding me!

He then proceeded to replace the entire 1,500 lines and spurious lookup tables with half a line of code.

Let’s put that into an equation containing hypothetical numbers:

  • For the sake of the process, let’s assume test and release amounts are equivalent
  • OEM charges $1,000 per day for a dev
  • Third-party charges $900 per day for a dev
  • OEM developer (who knows how the OEM software works) takes 15 seconds to write the code  = $0.52
  • Third-party dev takes (conservatively) 5 days to write the equivalent code (which didn’t work properly) = $4,500

In the grand scheme of this multi-million dollar project, the additional $4,499.48 was almost meaningless, but it introduced weeks of delays (diagnosis, re-dev, re-test, re-release, etc).

Now, let’s say the new functionality offered by this code was worth $50,000 to the carrier in efficiency gains. Who deserves to be rewarded $5,000 for value delivered?

  • The OEM who completed the task and got it working in seconds (and was compensated $0.52); or
  • The Third-party who never got it to work despite a week of trying (and was compensated $4,500)

The hard part about scope of works projects is that someone has to scope them and define the value delivered by them. That’s a whole lot harder and provides less flexibility than just assigning a day rate. But perhaps that in itself provides value. If we need to consider the value of what we’re producing, we might just find that some of the tasks in our agile backlog aren’t really worth doing.

If you’d like to share your thoughts on this, please leave a comment below.

 

 

 

The confused mind says no – the psychology of OSS purchasing

When it comes to OSS/BSS vendor selections, a typical customer might say they’re evaluating vendors on criteria that are a mix of technical and commercial (and other) factors. However, there’s more likely a much bigger and often hidden factor that drives a purchasing event. We’ll explore what that is shortly.

I’m currently reading the book, Alchemy: The Surprising Power of Ideas That Don’t Make Sense. It discusses a number of lateral, counter-intuitive approaches that have actually been successful and the psychological effects behind them.

The author, Rory Sutherland, proffers the idea that, “we make decisions… not only for the expected average outcome (but) also seek to minimise the possible variance, which makes sense in an uncertain world.” Also that, “A 1 percent chance of a nightmarish experience dwarfs a 99 percent chance of a 5 percent gain.”

Does that make you wonder about what the OSS/BSS buying experience feels like?

Are OSS/BSS vendors so busy promoting the 5% gain and the marginal differentiating features that they’re overlooking the white-knuckled fear of the nightmarish experience for OSS buyers?

OSS/BSS transformation projects tend to be large, complex and risky. There are a lot of unknowns, especially for organisations that tackle these types of projects rarely. Every project is different. The stakeholders signing off these projects are making massive investment decisions (relative to their organisation’s respective size) in the allocation of  resources (in financial, human and time allocation). The ramifications of these buying decisions will last for years and can often be career-defining (in the positive or the negative depending on the success of the transformation).

As someone who assists organisations with their buying decisions, I can concur with the old saying that, “the confused mind says no.” I’d also suggest that the scared mind says F#$@ no! If the vendor bamboozles the buyer with jargon and features and data, it only amplifies the fears that they might be walking into a nightmarish experience.

Fear and confusion are the reason customers often seek out the vendors who are in the top-right corner of the Gartner quadrant, even when they’re just not the best-fit solution. It’s the reason for the old saying that nobody got fired for hiring IBM. It’s the reason OSS/BSS procurement events can be so arduous (9, 12, 18 months are the norm).

The counter-intuitive approach for vendors is to spend more time overcoming the fear and confusion rather than technical demonstrations:

  • Simplify the messaging
  • Simplify the user experience (refer to the OSS intuition age)
  • Simplify the transformation
  • Provide work breakdowns and phasing to deliver early proof of value rather than a big-bang delivery way off into the future
  • Taking time to learn and communicate in the customer’s voice and terminology rather than language that’s more comfortable to you
  • Provide working proofs-of-concept / sandpits of your solutions as early as possible for the customer to interact with
  • Allow customers to use these sandpit environments and self-help with extensive support collateral (eg videos, how-to’s) enabling the customer to build trust in you at their own pace
  • Show evidence of doing the important things really well and efficiently rather than a long-tail of marginal features
  • Show evidence of striving to ensure every customer gets a positive outcome. This includes up-front transparency of the challenges faced (and still being faced) on similar projects. Not just words, but evidence of your company’s actions on behalf of customers. This might include testimonials and referrals for every single customer
  • Show evidence of no prior litigations or rampant variations or cost escalations on past projects
  • Trust is required to reduce fear and confusion (refer to “the relationship slider” in the three project responsibility sliders)
  • Provide examples of past challenges and risk mitigations. Even school the client on what they need to do to de-risk the project prior to commencement*

Can you think of other techniques / proofs that almost guarantee to the customer that they aren’t entering into a nightmarish situation?

* Note: I wrote Mastering your OSS with this exact concept in mind – to get customers ready for the transformation project they’re about to embark on and the techniques they can use to de-risk the project.

 

Are you kidding? We’ll never use open-source OSS/BSS!

Back in the days when I first started using OSS/BSS software tools, there was no way any respectable telco was going to use open-source software (the other oss, for which I’ll use lower-case in this article) in their OSS/BSS stacks. The arguments were plenty, and if we’re being honest, probably had a strong element of truth in many cases back then.

These arguments included:

  • Security – This is the most commonly cited aversion I’ve heard to open-source. Our OSS/BSS control our network, so they absolutely have to be secure. Secure across all aspects of the stack from network / infrastructure to data (at rest and in motion) to account access to applications / code, etc. The argument against open-source is that the code is open to anyone to view, so vulnerabilities can be identified by hackers. Another argument is that community contributors could intentionally inject vulnerabilities that aren’t spotted by the rest of the community
  • Quality – There is a perception that open-source projects are more hobby projects  than professional. Related to that, hobbyists can’t expend enough effort to make the solution as feature-rich and/or user-friendly as commercial software
  • Flexibility – Large telcos tend to want to steer the products to their own unique needs via a lot of customisations. OSS/BSS transformation projects tend to be large enough to encourage proprietary software vendors to be paid to make the requested changes. Choosing open-source implies accepting the product (and its roadmap) is defined by its developer community unless you wish to develop your own updates
  • Support – Telcos run 24x7x365, so they often expect their OSS/BSS vendors to provide round-the-clock support as well. There’s a  belief that open-source comes with a best-effort support model with no contracted service obligations. And if something does go drastically wrong, that open-source disclaims all responsibility and liability
  • Continuity – Telcos not only run 24x7x365, but also expect to maintain this cadence for decades to come. They need to know that they can rely on their OSS/BSS today but also expect a roadmap of updates into the future. They can’t become dependent upon a hobbyist or community that decides they don’t want to develop their open-source project anymore

Luckily, these perceptions around open-source have changed in telco circles in recent years. The success of open-source organisations like Red Hat (acquired by IBM for $34 billion on annual revenues of $3.4 billion) have shown that valuable business models can be underpinned by open-source. There are many examples of open-source OSS/BSS projects driving valuable business models and associated professionalism. The change in perception has possibly also been driven by shifts in application architectures, from monolithic OSS/BSS to more modular ones. Having smaller modules has opened the door to utilisation of building block solutions like the Apache projects.

So let’s look at the same five factors above again, but through the lens of the pros rather than the cons.

  • Security – There’s no doubt that security is always a challenge, regardless of being open-source or proprietary software, especially for an industry like OSS/BSS where all organisations are still investing more heavily in innovation (new features/capabilitys) more than security optimisations. Clearly the openness of code means vulnerabilities are more easily spotted in open-source than in “walled-garden” proprietary solutions. Not just by nefarious actors, but its development community as well. Linus’ Law suggests that “given enough eyeballs, all bugs (and security flaws) are shallow.” The question for open-source OSS/BSS is whether there are actually many eyeballs. All commercially successful open-source OSS/BSS vendors that I’m aware of have their own teams of professional developers who control every change to the code base, even on the rare occasions when there are community contributions. Similarly, many modern open-source OSS/BSS leverage other open-source modules that do have many eyes (eg linux, snmp libaries, Apache projects, etc). Another common perception is security through obscurity, that there are almost no external “eyeballs.” The fragmented nature of the OSS/BSS industry means that some proprietary tools have a tiny install base. This can lull some into a false sense of security. Alternatively, open-source OSS/BSS manufacturers know there’s a customer focus on security and have to mitigate this concern. The other interesting perspective of openness is that open-source projects can quite quickly be scrutinised for security-code-quality. An auditor has free reign to identify whether the code is professional and secure. With proprietary software, the customer’s auditor isn’t afforded the same luxury unless special access is granted to the code. With no code access, the auditor has to reverse-engineer for vulnerabilities rather than foresee them in the code.
  • Quality – There’s no doubt that many open-source OSS/BSS have matured and found valuable business models to sustain them. With the profitable business model has come increased resources, professionalism and quality. With the increased modularity of modern architectures, open-source OSS/BSS projects are able to perform very specific niche functionalities. Contrast this with the monolithic proprietary solutions that have needed to spread their resources thinner across a much wider functional estate. Also successful open-source OSS/BSS organisations tend to focus on product development and product-related services (eg support), whereas the largest OSS/BSS firms tend to derive a much larger percentage of revenues from value-added services (eg transformations, customisations, consultancy, managed services, etc). The latter are more services-oriented companies than product companies. As inferred in the security point above, open-source also provides transparency relating to code-quality. A code auditor will quickly identify whether open-source code is of good quality, whereas proprietary software quality is hidden inside the black-box 
  • Flexibility – There has been a significant shift in telco mindsets in recent years, from an off-the-shelf to a build-your-own OSS/BSS stack. Telcos like AT&T have seen the achievements of the hyperscalers, observed the increased virtualisation of networks and realised they needed to have more in-house software development skills. Having in-house developers and access to the code-base of open-source means that telcos have (almost) complete control over their OSS/BSS destinies. They don’t need to wait for proprietary vendors to acknowledge, quote, develop and release new feature requests. They no longer rely on the vendor’s roadmap. They can just slip the required changes into their CI/CD pipeline and prioritise according to resource availability. Or if you don’t want to build a team of developers specifically skilled with your OSS/BSS, you can pick and choose – what functionality to develop in-house, versus what functionality you want to sponsor the open-source vendor to build
  • Support – Remember when I mentioned above that OSS/BSS organisations have found ways to build profitable business models around open-source software? In most cases, their revenues are derived from annual support contracts. The quality and coverage of their support (and the products that back it up) is directly tied to their income stream, so there’s commensurate professionalism assigned to support. As mentioned earlier, almost all the open-source OSS/BSS I’m aware of are developed by an organisation that controls all code change, not community consensus projects. This is a good thing when it comes to support, as the support partner is clear, unlike community-led open-source projects. Another support-related perspective is in the number of non-production environments that can be used for support, test, training, security analysis, etc. The cost of proprietary software means that it can be prohibitive to stand up additional environments for the many support use-cases. Apart from the underlying hardware costs and deployment effort, standing up additional open-source environments tends to be at no additional cost. Open-source also gives you greater choice in deciding whether to self-support your OSS/BSS in future (if you feel that your internal team knows the solution so intimately that they can fix any problem or functionality requirement that arises) rather than paying ongoing support contracts. Can the same be said for proprietary product support?
  • Continuity – This is perhaps the most interesting one for me. There is the assumption that big, commercial software vendors are more reliable than open-source vendors. This may (or might not) be the case. Plenty of commercial vendors have gone out of business, just as plenty of open-source projects have burned out or dwindled away. To counter the first risk, telcos pay to enter into software escrow agreements with proprietary vendors to ensure critical fixes and roadmap can continue even in the event that a vendor ceases to operate. But the escrow contract may not cover when a commercial vendor chooses to obsolete a product line of software or just fail to invest in new features or patches. This is a common event from even the world’s largest OSS/BSS providers. Under escrow arrangements, customers are effectively paying an insurance fee to have access to the code for organisational continuity purposes, not product continuity. Escrow may not cover that, but open-source is available under any scenario. The more important continuity consideration is the data and data is the reason OSS/BSS exist. When choosing a commercial provider, especially a cloud software / service provider, the data goes into a black box. What happens to the data inside the black box is proprietary and often what comes out of it is also. Telcos will tend to have far more control of their data destinies for operational continuity if using open-source solutions. Speaking of cloud and SaaS-model and/or subscription-model OSS/BSS, customers are at the whim of the vendor’s product direction. Products, features and modules can be easily changed or deprecated by these vendors, with little recourse for the buyers. This can still happen in open-source and become a impediment for buyers too, but at least open-source provides buyers with the opportunity to control their own OSS/BSS destinies.

Now, I’m not advocating one or the other for your particular situation. As cited above, there are clearly pros and cons for each approach as well as different products of best-fit for different operators. However, open-source can no longer be as summarily dismissed as it was when I first started on my OSS/BSS journey. There are many fine OSS and BSS products and vendors in our Blue Book OSS/BSS Vendor Directory that are worthy of your consideration too when looking into your next product or transformation.

Edit: Thanks to Guy B. who rightly suggested that Scalability was another argument against open-source in the past. Ironically, open-source has been a significant influencer in the almost unlimited scalability that many of our solutions enjoy today.

There’s an OSS Security Elephant in the Room!

The pandemic has been beneficial for the telco world in one way. For many who weren’t already aware, it’s now clear how incredibly important telecommunications providers are to our modern way of life. Not just for our ability to communicate with others, but our economy, the services we use, the products we buy and even more fundamentally, our safety.

Working in the telco industry, as I’m sure you do, you’ll also be well aware of all the rhetoric and politics around Chinese manufactured equipment (eg Huawei) being used in the networks of global telco providers. The theory is that having telecommunications infrastructure supplied by a third-party, particularly a third-party aligned with non-Western allies, puts national security interests at risk.

In this article, “5G: The outsourced elephant in the room,” Bert Hubert provides a brilliant look into the realities of telco network security that go far beyond just equipment supply. It breaks the national security threat into three key elements:

  • Spying (using compromised telco infrastructure to conduct espionage)
  • Availability (compromising and/or manipulating telco infrastructure so that it’s unable to work reliably)
  • Autonomy (being unable to operate a network or to recover from outages or compromises)

The first two are well understood and often discussed. The third is the real elephant in the room. The elephant OSS/BSS have a huge influence over (potentially). But we’ll get to that shortly.

Before we do, let’s summarise Bert’s analysis of security models. For 5G, he states that there’s an assumption that employees at national carriers design networks, buy equipment, install it, commission it and then hand it over to other employees to monitor and manage it. Oh, and to provide other specialised activities like lawful intercept, where a local legal system provides warrants to monitor the digital communications of (potentially) nefarious actors. Government bodies and taxpayers all assume the telcos have experienced staff with the expertise to provide all these services.

However, the reality is far different. Service providers have been outsourcing many of these functions for decades. New equipment is designed, deployed, configured, maintained and sometimes even financed by vendors for many global telcos. As Bert reinforces, “Just to let that sink in, Huawei (and their close partners) already run and directly operate the mobile telecommunication infrastructure for over 100 million European subscribers.

But let’s be very clear here. It’s not just Huawei and it’s not just Chinese manufacturers. Nor is it just mobile infrastructure. It’s also cloud providers and fixed-line networks. It’s also American manufacturers. It’s also the integrators that pull these networks and systems together. 

Bert also points out that CDRs (Call Detail Records) have been outsourced for decades. There’s a strong trend for billing providers to supply their tools via SaaS delivery models. And what are CDRs? Only metadata. Metadata that describes a subscriber’s activities and whereabouts. Data that’s powerful enough to be used to assist with criminal investigations (via lawful intercept). But where has CDR / bill processing been outsourced to? China and Israel mostly.

Now, let’s take a closer look at the autonomy factor, the real elephant in the room. Many design and operations activities have been offshored to jurisdictions where staff are more affordable. The telcos usually put clean-room facilities in place to ensure a level of security is applied to any data handled off-shore. They also put in place contractual protection mechanisms.

Those are moot points, but still not the key point here. As Bert brilliantly summarises,  “any worries about [offshore actors] being able to disrupt our communications through backdoors ignore the fact that all they’d need to do to disrupt our communications.. is to stop maintaining our networks for us!

There might be an implicit trust in “Western” manufacturers or integrators (eg Ericsson, Nokia, IBM) in designing, building and maintaining networks. However, these organisation also outsource / insource labour to international destinations where labour costs are cheaper.

If the R&D, design, configuration and operations roles are all outsourced, where do the telcos find the local resources with requisite skills to keep the network up in times when force majeure (eg war, epidemic, crime, strikes, etc) interrupts a remote workforce? How do local resources develop the required skills if the roles don’t exist locally?

Bert proposes that automation is an important part of the solution. He has a point. Many of the outsource arrangements are time and materials based contracts, so it’s in the resource suppliers’ best interests for activities to take a lot of time to maintain manually. He counters by showing how the hyperscalers (eg Google) have found ways of building automations so that their networks and infrastructure need minimal support crews.

Their support systems, unlike the legacy thinking of telco systems, have been designed with zero-touch / low-touch in mind.

If we do care about the stability, resiliency and privacy of our national networks, then something has to be done differently, vastly different! Having highly autonomous networks, OSS, BSS and related systems is a start. Having a highly skilled pool of local resources that can research, design, build, commission, operate and improve these systems would also seem important. If the business models of these telcos can’t support the higher costs of these local resources, then perhaps national security interests might have to subsidise these skills?

I wonder if the national carriers and/or local OSS/BSS / Automation suppliers are lobbying this point? I know a few governments have inserted security regulatory systems and pushed them onto the telcos to adhere to, to ensure they have suitable cyber-security mechanisms. They also have lawful intercept provisions. But do any have local operational autonomy provisions? None that I’m aware of, but feel free to leave us a comment about any you’re aware of.

PS. Hat tip to Jay for the link to Bert’s post.

How to make your OSS a Purple Cow

With well over 400 product suppliers in the OSS/BSS market it can be really difficult to stand out from the other products. Part of the reason we compiled The Blue Book OSS/BSS Vendor Directory was to allow us to quickly recall one product from another. With so much overlapping functionality and similarities in their names, some vendors can “blend” into each other when we try to recall them.

And we spend a lot of our week working with and analysing the market, the products and the customers who use them. Imagine how difficult that task would be for someone whose primary task is to operate a network (which most OSS/BSS customers do for a living).

How then can a vendor make their offerings stand out amongst this highly fragmented product market?

Seth Godin is a legendary marketer and product maker (not in OSS or BSS products though). We refer to him often here on the Passionate About OSS blog because of his brilliant and revolutionary ideas. One of those ideas turned into a product manifesto entitled Purple Cow. This book made it into our list of best books for OSS/BSS practitioners.

Purple Cow describes something phenomenal, something counterintuitive and exciting and flat-out unbelievable… Seth urges you to put a Purple Cow into everything you build, and everything you do, to create something truly noticeable.”

When you’re on a long trip in the countryside, seeing lots of brown or black cows soon gets boring, but if you saw a purple cow, you’d immediately take notice. This book provides the impetus to make your products stand out and drive word of mouth rather than having to differentiate via your marketing.

I’ve noticed the same effect when we pitch our Managed OSS/BSS Data Service to prospects. It’s the data collection and collation tools that drive most of the real business value (in our humble opinion), but it’s the visualisation tools that drive the wow factor amongst our prospects / customers.

It’s our ability to show 3D models of assets (eg towers as per the animation below) or overlay of real-time data onto the 3D models (ie digital twins), or mashing up many disparate data sources for presentation by powerful and intuitive visualisation engines.

This might seem counter-intuitive, but set aside your products’ technical functionality for a moment. Now ask yourself what is the biggest wow factor that stays with your customers and gets them talking with others in the industry? What’s phenomenal, counterintuitive, exciting and flat-out unbelievable? What’s your Purple Cow?

Having reviewed many OSS/BSS products, I can assure you there aren’t many Purple Cows in the OSS/BSS industry. If you’re responsible for products or marketing at an OSS/BSS vendor, that gives you a distinct opportunity.

Improvements in technologies such as 3D asset modelling, AI image recognition, advances in UX/CX, data collection / visualisation and many more gives you the tools to be creative. The tools to be memorable. The tools to build a Purple Cow.

It’s difficult to stand out based on functional specifications or non-functionals (unless you leave others in your dust, such as being able to ingest data at 10-20x your nearest competitor). Those features might be where the your biggest business value may lie. In fact that’s probably where it does lie. 

However, it seems that the Purple Cows in OSS/BSS appear in the unexpected and/or surprising visual experiences.

Can you imagine building an OSS/BSS Purple Cow? If interested, we’d be delighted to assist.

 

Launch of The Passionate About OSS Podcast

We’re excited to announce the Launch of The Passionate About OSS Podcast.

The first batch of five episodes can be found here, with new episodes to be released here on a weekly basis:

The Passionate About OSS Podcast

The aim of the show is to shine a light on the many brilliant people who work in the OSS industry.

We’ll interview experts in the field of OSS/BSS and telecommunications software. Guests represent the many facets of OSS including: founders, architects, business analysts, designers, developers, rainmakers, implementers, operators and much more, giving a 360 degree perspective of the industry.

We’ll delve into the pathways they’ve taken in achieving their god-like statuses, but also unlock the tips, tactics, methodologies and strategies they employ. Their successes and failures, challenges and achievements. We’ll look into the past, present and even seek to peer into what the future holds for the telco and OSS industries.

OSS/BSS in the clouds, updated

We posted an article in July entitled “OSS / BSS in the clouds,” which looked at the OSS, BSS and related telco infrastructure platforms being offered by AWS, Google, Microsoft and their partners. This followed a number of recent announcements made by the hyperscalers relating to their bigger pushes into telco. It had a particular focus on 5G models and the edge compute technologies that support them. 

Following VMware’s announcement of its 5G Telco Cloud Portfolio on 1st Sept 2020, we’ve now retro-fitted VMware into the earlier post. Click on it here “OSS / BSS in the clouds,” to view the updated article.

Is omni-channel more disadvantage than advantage for telcos?

In our post on Monday, we discussed how some commodity providers have a structural advantage through lower cost of production (eg Rio Tinto in iron ore). Telcos have the potential to achieve a similar advantage on their commodity services too.

It also mentioned that the first principle behind that advantage is simplicity (of systems, overheads, processes, offerings, etc). Many of these simplification factors are controllable by our OSS/BSS, but most just end up with us as a result of up-stream decisions that flow to us for resolution. These decisions (eg running 50+ mobile offering variants including grandfathered services) make our OSS/BSS far more complex.

Another example is omni-channel customer engagement, which includes:

  • Call centres
  • Digital / websites
  • Email
  • SMS
  • Chatbots
  • Mobile Apps
  • IVR (Interactive Voice Response)
  • Social channels
  • Smart-home assistants
  • USSD (Unstructured Supplementary Service Data)

I completely get that we want to allow customers (or potential customers) to choose the contact mechanism/s they’re most comfortable with. Tight coupling / orchestration across these channels is important for customer loyalty.

Unfortunately, the task of coordinating all of these systems is complex. Linking and tracking customer journeys through multiple channels is even more challenging. For example, websites and IVR channels don’t require customers to self-identify, making it difficult to cross-link with in app touch-point data where identification is inherent. Some flows are personalised, some are interactive, some are batched. 

Gaining consolidated insights from these disparate data sources is imperative for customer satisfaction. 

Clearly a typical telco omni-channel story doesn’t represent simplicity, especially for the OSS/BSS that support it! That presents a barrier to achieving the cost of production advantage we discussed.

How do telcos compare with Apple, Amazon, Google and others? Do they offer as many customer contact channels? Do they have as many variants in their “extended supply chain?” Do they have as many potential fall-out points in their O2A, T2R, P2O, etc workflows? It appears to me that most telcos are at a structural disadvantage here (on costs and CX).

We’d love to hear your thoughts. Please leave us your perspectives in the comments section below.  

Is scaled OSS/BSS multi-tenancy a thing?

We talked yesterday about the commoditisation of telco services and the part that OSS/BSS have to play in differentiation. We also talked about telcos retaining a few competitive advantages despite the share-of-wallet inroads made by OTT, software and cloud service providers recently. Managed services is one area where some of those advantages converge.

Quite a few years ago I worked with one of Australia’s largest telcos on a managed service account for one of Australia’s big four banks. The value of the managed service contract was worth a couple of billion dollars. It covered the whole gamut of what a telco can offer. Voice, data, mobility and unified communications of course, but also branch / office comms fit-outs, international WANs, IT services, security, custom help-desk, dedicated service delivery management and much more.

Big companies tend to prefer to deal with big companies, especially when the services are complex and wide-ranging. A bank like this one could realistically only negotiate with a handful of telcos because only a few could provide viable size and scope to meet their needs… if they wanted to keep it to a consolidated procurement event.

This bank wasn’t this telco’s only large managed services contract. They had quite a few similar clients. The deals were all incredibly cut-throat between the other big telcos, but they were still profitable, especially once the variations started coming in. The telcos are not in a race to the bottom on managed services like the commodity data services we discussed yesterday.

I found it interesting that the telcos mainly focused on providing these managed service deals to the ASX200 (Australia’s top 200 companies by market capitalisation), give or take. So they service the top 200 (ish) with managed services and the long, long tail with retail services. But of course mid-market companies fall somewhere in the middle – big enough to require custom solutions, but too small to warrant the type of managed services the bank was getting.

So, now let me put the OSS/BSS spin on this situation (yes, it took me a while). At this telco, each of its big managed service contracts had completely bespoke OSS/BSS/portal solutions. Of course they had a core OSS/BSS – the factory that handled assurance and fulfilment for customers large and small – but every managed service had completely unique, satellite OSS/BSS/portals at their customer interface. For example, the core ran eTOM, the customer interface often demanded ITIL.

This telco couldn’t offer managed services to the mid-market because it was too expensive to set up the unique tooling. That meant there were almost no repeatability or economies of scale in processes, reporting, data science, skill portability, etc.

It bewildered me that they hadn’t invested the time into creating a cookie-cutter approach to all these satellite OSS/BSS/portals. Sure, each would need some customisation, but there was also significant potential for repeatability / consistency.

Now this may sound a bit bewildering to you too. But it’s not just the telco’s short-term thinking that leads to this situation. Most OSS/BSS vendors don’t build products with this type of multi-tenancy in mind.

The ability to spin up/down satellite OSS/BSS/portals, each with configurability but consistency, is not common. Importantly, nor is secure partitioning that ensures each customer only sees their data even when using shared infrastructure (including NOC / WOC / SOC as well as network infra and core OSS/BSS).

You know, I don’t recall ever hearing anyone else talk about this type of use-case. Multi-tenancy, yes, to an extent, but not at managed services scale.

Is this even a thing? Should it be? Could it be? You be the judge! I’d love to hear your thoughts / experiences. Please leave a comment below if your OSS/BSS stack supports this use-case or if your telco has developed a cookie-cutter approach. What approach do you use?

OSS’s Influence on Cost of Production

Since widespread deregulation of telecommunications globally, the passing of data has become a commodity. Perhaps it always was, but increased competition has steadily driven down dollar per bit. It’s likely to continue on that path too. Meanwhile the expected throughputs and consumption of data services is ramping ever-upwards, which requires investment in networks by their operators.

By definition, a commodity is a product/service that is indistinguishable between providers. The primary differentiator is price because there are no other features that make a buyer choose one provider over another. 

At face value, that’s true of commodities such as oil or iron ore, just as it is for data. What could be less differentiated than ones and zeroes?

However, as the charts below show for oil and iron ore, commodities aren’t always a level playing field. Some suppliers have significant differentiation – via cost of production advantages over their competitors.

Cost of Oil Production

Iron ore cost of Production

What if we created a $/bit graph of telcos similar to the oil graph above showing opex and capex splits? Which countries / operators would have significant advantage? Which telcos are like Rio Tinto, comfortable in the knowledge that no matter how low the spot price goes, their competitors will be deeply unprofitable (and possibly out of business) long before they will be.

Virtualisation of network infrastructure (SDN, NFV, cloud) has been the much-hyped saviour of telco business models. However, like me, you’ve probably started hearing news that savings from these architectures just aren’t eventuating. Cost-bases are possibly even going up because of their increased complexity. Either way, incremental cost reduction tends to have diminishing returns anyway.

It would seem that telcos are left with two / three choices:

  1. Have structurally lower production costs, like Kuwait and Rio Tinto; or
  2. Differentiate / Innovate, limiting exposure to the raw data / connectivity services that so many telcos still rely upon
  3. Or both

OSS/BSS has a part to play with both of these options.

Structurally lower cost comes not from cost reduction but from having far simpler variant trees (ie smaller numbers of product offerings, network types, system integrations, service configuration styles, support models, obsolescence of legacy / grandfathered products, minimal process options, etc, etc). Some of this happens within the OSS/BSS, but a lot more of it stems from upstream decisions.

Differentiation / innovation means being able to innovate through experiences / insights / content / trust / partnerships / ecosystems / local-presence in ways that other organisations can’t. It’s unlikely to be in software alone or in cloud infrastructure because others have proven to do that far more effectively than telcos. As much as they wish otherwise, it’s just not in the DNA of many. Yet that’s where most attention seems to be. Meanwhile OSS/BSS are waiting to be the glue that can leverage the competitive advantages that telcos do still hold.

How fragmentation is harming the OSS/BSS industry

Our Blue Book OSS/BSS Vendors Directory provides a list of over 400 vendors. That clearly states that it’s a highly fragmented market. This amount of fragmentation hurts the industry in many ways, including:

  • Duplication – Let’s say 100 of the 400 vendors offer alarm / fault management capabilities. That means there are 100 teams duplicating effort in creating similar functionality. Isn’t that re-inventing the wheel, again and again? Wouldn’t the effort be better spread into developing new / improved functionality, rather than repeating what’s already available (more or less). And it’s not just coding, but testing, training, etc. The talent pool overlaps on duplicated work at the expense of taking us faster into an innovative future
  • Profit-share – The collective revenues of all those vendors need to be spread across many investors. Consolidated profits would most likely lead to more coordination of innovation (and less duplication above). And just think at how much capability has been lost in tools developed by companies that are no longer commercially viable
  • Overhead – Closely related is that every one of these organisations has an overhead that supports the real work (ie product development, project implementation, etc). Consolidation would bring greater economies of scale
  • Consistency – With 400+ vendors, there are 400+ visions / designs / architectures / approaches. This means the cost and complexity of integration hurts us and our customers. The number of variants makes it impossible for everything to easily bolt together – not withstanding the wonderful alignment mechanisms that TM Forum, MEF, etc create (via their products Frameworx, OpenAPIs, MEF 3.0 Framework, LSO, etc). At the end of the day, they create recommendations that vendors can interpret as they see fit. It seems the integration points are proliferating rather than consolidating
  • Repeatability and Quality – Repeatability tends to provide a platform for continual improvement. If you do something repeatedly, you have more opportunities to refine. Unfortunately, the bespoke nature of OSS/BSS implementations (and products) means there’s not a lot of repeatability. Linus’s Law of OSS defects also applies, with eyeballs spread across many code-bases. And the spread of our variant trees means that we can never have sufficient test coverage, meaning higher end-to-end failure / fall-out rates than should be acceptable 
  • Portability – Because each product and implementation is so different, it can be difficult to readily transfer skills between organisations. An immensely knowledgeable, talented and valuable OSS expert at one organisation will likely still need to do an apprenticeship period at a new organisation before becoming nearly as valuable
  • Analysis Paralysis – If you’re looking for a new vendor / product, you generally need to consider dozens of alternatives. And it’s not like the decisions are easy. Each vendor provides a different set of functionality, pros and cons. It’s never a simple “apples-for-apples” comparison (although we at PAOSS have refined ways to make comparisons simpler). It’s certainly not like a cola-lover having to choose between Coke and Pepsi. The cost and ramifications of an OSS/BSS procurement decision are decidedly more significant too obviously
  • Requirement Spread – Because there are so many vendors with so many niches and such a willingness to customise, our customers tend to have few constraints when compiling a list of requirements for their OSS/BSS. As described in the Lego analogy, reducing the number of building blocks, perhaps counter-intuitively, can actually enhance creativity and productivity
  • Shared Insight – Our OSS/BSS collect eye-watering amounts of data. However, every data set is unique – collection / ETL approach, network under management, product offerings, even naming conventions, etc. This makes it challenging to truly benchmark between organisations, or share insights, or share seeded data for cognitive tools

However, I’m very cognisant that OSS come in all shapes and sizes. They all have nuanced requirements and need unique consideration. Yet many of our customers stand on a burning platform and desperately need us to create better outcomes for them.

From the points listed above, the industry is calling out for consolidation – especially in the foundational functionality that is so heavily duplicated – inventory / resource management, alarms, performance, workflows, service ordering, provisioning, security, infrastructure scalability, APIs, etc, etc. 

If we had a consistent foundation for all to work on, we could then more easily take the industry forward. It becomes a platform for continuous improvement of core functionality, whilst allowing more widespread customisation / innovation at its edges.

But who could provide such a platform and lead its over-arching vision?

  • I don’t think it can be a traditional vendor. Despite there being 400+ vendors, I’m not aware of any that cover the entire scope of TM Forum’s TAM map. Nor do any hold enough market share currently to try to commandeer the foundational platform
  • TM Forum, wouldn’t want to compromise their subscriber base by creating something that overlaps with existing offerings
  • Solution Integrators often perform a similar role today, combining a multitude of different OSS/BSS offerings on behalf of their customers. But none have a core of foundational products that they’ve rolled out to enough customers to achieve critical mass
  • I like the concept of what ONAP is trying to do to rally global carriers to a common cause. However, its size and complexity also worries me. That it’s a core component of the Linux Foundation (LF Networking) gives it more chance of creating a core foundation via collaborative means rather than dictatorial ones

We’d love to hear your thoughts. Is fragmentation a good thing or a bad thing? Do you suggest a better way? Leave us a message below.

How to Design Telecommunication Business Process Flows Using eTOM

Have you been tasked with designing process flows for a telecommunication network operator? Do these include end-to-end (E2E) processes that leverage one (or likely more) of your OSS/BSS tools along the journey? Perhaps you’ve even been tasked with setting a roadmap for OSS/BSS development and/or integration with other OSS/BSS?

This can be a daunting task because there are so many E2E processes that are required to run the business and operations of a network service provider. Where do you start?

Lucky for you and the rest of the industry, the TM Forum has provided a valuable set of tools that have become the benchmark for designing business processes for network operators globally. It’s known as the eTOM (enhanced Telecommunication Operations Map), which is part of the TM Forum’s Frameworx suite. The origins of eTOM began in the 1990s when the TM Forum sought to assist in the understanding of external business linkages for interface design.

Due to the unique nature and requirements of each organisation, eTOM was always intended to be flexible, but also provide as much standardisation of process as possible. This would provide consistency between vendors / integrators of telco software like OSS/BSS, but also consistency in the delivery of services across points of interconnect (POI) between the networks of different service providers.

The eTOM business process framework has since grown to include:

  • The main document (GB921) – A spreadsheet that contains a list of many (1,000+) atomic tasks (or decompositions) that can be used to create E2E flows from
  • Addendum D (GB921D) – A document that provides a hierarchical and functional grouping of tasks, with multiple levels of granularity (Level 0 is shown in the first diagram below)

    (then a decomposition down to task level 3 can be formed, as shown in the second diagram below. Note: don’t worry about the details here, as we’ll get into that later)
  • Addendum E (GB921E) – A document that describes the design of E2E business streams from the atomic tasks described in GB921. The diagram below comes directly from GB921E. The “customer centric processes” on the left panel represent customer-initiated workflows (more on those later in this article). Meanwhile, the bottom right corner shows how the atomic tasks are linked to form an E2E process for each customer centric process (using L3 decompositions in this example):
  • Addendum F (GB921F) – A document that provides examples of end to end business processes, built from the atomic tasks provided in the previously mentioned documents. The E2E samples include Request to Answer (also known as R2A), Order to Payment (O2P), Request to Change (R2C) and many others, which we mention in this article about key business process acronyms in the OSS/BSS industry
  • Addendum G (GB921G) – A document that provides a guide on how to apply the eTOM process framework
  • There are multiple other addenda that can be used to assist you with the development of your organisation’s E2E processes and integrations, including Addendum W (GP921W), which describes how ITIL (an IT process framework) and eTOM can work together

The important feature to understand here is that eTOM comprises:

  • A large list of tasks (GB921), which aren’t process flows in isolation, but need to be joined together as…
  • …Suggested sequences of tasks (eg GB921E, GB921F) to guide the creation of your E2E processes to meet your workflow requirements 

The art is in the building of E2E flows. The tasks are prescribed by eTOM. The processes only partially so. You have the option of forming them using the examples in GB921F, but you will probably also require input from your operational teams, business analysts, past experiences, inputs from vendors (eg OSS/BSS product functionality capabilities), etc. You’re left with infinite possibilities. So you might still be asking, where do I start?

Well, the industry tends to make E2E processes that are initiated by a customer (or internal operator / engineer). They start with that trigger (X) and end with the outcome (Y) that they’ve sought to trigger (ie closing the loop). They are often, but not always, named using the “X to Y” convention. eg, the link above, provides many examples of these, such as Order to Activate (O2A), Order to Cash (O2C), Trouble to Resolve (T2R), etc, etc, etc.

GB921E provides the following examples:

  • Customer-Centric E2E business streams:
    • Request to Answer
    • Order to Payment
    • Usage to Payment
    • Request to Change
    • Termination to Confirmation
    • Complaint to Solution
  • Network E2E business streams:
    • Production Order to Acceptance
    • Trouble Ticket to Solution
    • Activation to Usage Data
    • Capacity Management
    • Service Lifecycle Management
    • Resource Lifecycle Management
  • Product E2E business streams:
    • Idea to Business Plan
    • Idea to Business Proposal
    • Business Proposal to Launch
    • Assessment to Relaunch
    • Assessment to Retirement
    • Market Strategy to Campaign
  • Engaged Party Flows:

Let’s take a closer look at GB921E and how it helps to solve for the first process in the list above –  R2A (Request to Answer) – using eTOM mappings.

Let’s first start with a Level 2 breakdown / mapping of R2A:

Then, this can be used to guide the Level 3 breakdown, which looks more like the E2E R2A process we’re expecting:

GB921E even provides the template for showing detailed information about the R2A process, as follows:

Note that the examples provided above were from eTOM release 20-5 (which includes GP921E v20.0.1). However, the eTOM document libraries are being refined constantly, with major version releases a couple of times each year, so revert back to the TM Forum eTOM page for latest updates before embarking on your business process designs.

Note: We’ve developed a technique to document, benchmark and optimise operational processes directly from OSS/BSS activity logs. This helps you capture current-state process flows in BPMN format to assist with your eTOM / ITIL process mapping exercise (see sample below).

Good luck on your journey of designing telco business processes for your organisation. If you require assistance, please don’t hesitate to contact us via the contact form below.

 

OSS / BSS in the clouds

Have you noticed the recent up-tick in headlines around telco offerings by hyperscalers AWS, Google and Microsoft? Or the multi-cloud telco models, the middleware, supplied by VMware and Red Hat?

Whilst previous generations of wireless connectivity have focussed on voice and data capabilities, 5G is architected to better enable consumer business models. Edge compute (both on-prem / device-edge and provider edge) and related services to support 5G use-cases appears to be the leading driver behind recent announcements.  These use-cases will need to be managed by our OSS/BSS for the telco operators and their customers.

Meanwhile, top-tier OSS/BSS users are also continuing to adopt cloud-native OSS/BSS initiatives, as described in this Infographic from Analysis Mason / Amdocs. Analysis Mason estimates that over 90% of CSPs in North America, Asia–Pacific and Europe will have their OSS/BSS stacks running on cloud infrastructure by 2022, with well over 60% on hybrid cloud.

However, just how much of the CSP OSS/BSS stack will be on the cloud remains in question. According to TM Forum’s research, most CSPs have deployed less than 5% percent of their operations software in the public cloud.

In today’s article, we take a closer look into cloud offerings for OSS/BSS. The providers we’ll cover are hyperscalers:

AWS

https://aws.amazon.com/telecom/

The following diagrams come from the Amazon Telco Symposium. The first diagram shows the AWS Telecom Engagement Model (noting the OSS/BSS bubble).

The latter diagram provides some insight into important offerings in AWS’ push into the 5G / telco edge such as Greengrass, SiteWise, Sagemaker and more.

 

AWS services such as the following have been used as part of home-grown offerings for years:

  • Wavelength (low latency), Lambda (serverless) or EC2 – compute services for processing applications/code
  • S3, EFS, Glacier, Elastic, Snow Family, etc – data storage for collecting logs, etc
  • EKS or ECS – for Kubernetes / Docker container / cluster management
  • VPC – for separate environment deployments
  • VPN – to tie VPCs to networks / clouds / DCs
  • ELB – for load balancing
  • ELK – for log management consisting of three open source projects: Elasticsearch, Logstash, and Kibana
  • Aurora, RDS, Redshift, DynamoDB, Neptune, KDB, etc – databases
  • Cassandra, Kibana, etc – data visualisation
  • SageMaker, Augmented AI, Lex, etc – AI / ML tools
  • And much more

These have been leveraged by telco architects to build home-grown OSS/BSS tools that leverage commercial and open-source products like Apache’s Kafka, NiFi, Spark, etc.

However, there’s been an increasing trend for OSS/BSS vendors to publish their offerings on the AWS marketplace too, including:

  • Moogsoft
  • Zoho / ManageEngine (eg OpManager, Network Config Mgr)
  • Solarwinds
  • Domotz Pro
  • Lumeta CloudVisibility
  • Flowmon
  • Hyperglance
  • Mphasis InfraGraf (Forecasting & Planning)
  • KloudGin (Work Order Mgmt)
  • Kx (network performance)
  • AND Bosch, ThingsBoard, ThingPark, ThingLogix, etc, etc (if we extend into IoT device management)

AWS Marketplace tends to show the solutions that are more standardised / fixed-price in nature (Telecoms section in Marketplace). Many other OSS/BSS vendors such as Netcracker, CSG, Intraway and Camvio don’t appear in the AWS marketplace but have customisable, AWS-ready solutions for clients. These companies have their own sales arms obviously, but also train the AWS global salesforce in their products.

Google Cloud

https://cloud.google.com/solutions/telecommunications

According to Google Cloud’s strategy for the telecom industry, Google Cloud is focusing on three strategic areas to support telecommunications companies:

  • Helping telecommunications companies monetise 5G as a business services platform, including:

    • The Global Mobile Edge Cloud (GMEC) strategy, which will deliver a portfolio and marketplace of 5G solutions built jointly with telecommunications companies; an open cloud platform for developing network-centric applications; and a global distributed edge for deploying these solutions

    • Anthos for Telecom, which will bring its Anthos cloud application platform to the network edge, allowing telecommunications companies to run their applications wherever it makes the most sense. Anthos for Telecom—based on open-source Kubernetes—will provide an open platform for network-centric applications.
  • Empowering telecommunications companies to better engage their customers through data-driven experiences by:

    • Empowering telecommunications companies to transform their customer experiences through data- and AI-driven technologies. Google’s BigQuery platform provides a scalable data analytics solution—with machine learning built-in so telecommunications companies can store, process, and analyze data in real time, and build personalization models on top of this data

    • Contact Center AI assists telecommunications companies with customer service. Contact Center AI gives companies 24/7 access to conversational self-service, with seamless hand-offs to human agents for more complex issues. It also empowers human agents with continuous support during their calls by identifying intent and providing real-time, step-by-step assistance
    • AI and retail solutions including omni-channel marketing, sales and service, personalisation and recommendations, and virtual-agent presence in stores
  • Assisting them in improving operational efficiencies across core telecom systems. This allows operators to move OSS, BSS and network functions from their own environments to the Google Cloud

This LightReading report even highlights how Google has been engaged to provide extensive knowledge transfer to some telcos.

This press release from March 2020 announced that Google would partner with Amdocs to support the telecom industry to:

  • Deliver Amdocs solutions to Google Cloud: Amdocs will run its digital portfolio on Google Cloud’s Anthos, enabling communications service providers (CSPs) to deploy across hybrid and multi-cloud configurations
  • Develop new enterprise-focused 5G edge computing solutions: Amdocs and Google Cloud will create new industry solutions for CSPs to monetize over 5G networks at the edge
  • Help CSPs leverage data and analytics to improve services: Amdocs will make its Data Hub and Data Intelligence analytics solutions available on Google Cloud. Amdocs and Google Cloud will also develop a new, comprehensive analytics solution to help CSPs leverage data to improve the reliability of their services and customer experiences.
  • Partner on Site Reliability Engineering (SRE) services: The companies will share tools, frameworks, and best practices for SRE and DevOps

On the same day the Google / Amdocs partnership was announced, Netcracker Technology announced it would deploy its entire Digital BSS/OSS and Orchestration stack on Google Cloud. These applications are cloud native, deployed as a set of reusable microservices that run over on-prem or public cloud on top of container platforms such as Google Kubernetes Engine (GKE).

Optiva (formerly Redknee) has also adopted a Google Cloud strategy, using Google Cloud Spanner and the Google Cloud Platform (GCP) to underpin its Charging Engine.

Ericsson has also announced a partnership with Google Cloud, to jointly develop 5G and edge cloud solutions as well as enterprise applications at the edge. Similarly, Nokia has also partnered with Google Cloud to collaborate on RAN / Open RAN, Cloud RAN and edge cloud technology / apps.

This article from CIMI Corp provides some great additional reading about why Google is a credible telco cloud provider.

Microsoft Azure

https://www.microsoft.com/en-us/industry/telecommunications

Microsoft has also announced an intention to better serve telecom operators at the convergence of cloud and comms networks through its Azure platform.

The diagram below, from this Microsoft blog, shows their coverage map of offerings for CSP customers (Azure for Operators):

Microsoft's Telco Offering Coverage Map

The blog also indicates that their offering is built upon:

  • Interconnect – 170 points of presence and 20,000 peering connections globally. More than 200 operators are already integrated with the Azure network via their ExpressRoute service
  • Edge – offered by the Azure platform, whether at enterprise edge, network edge, network core or cloud
  • Network Functions – This is where Microsoft distinguishes itself from AWS and Google. Its ability to offer network, particularly for 5G via RAN and Mobile Packet Core offerings (see more about Microsoft’s Affirmed and Metaswitch acquisitions below)
  • Cloud – Incorporates a marketplace with capabilities including OSS/BSS, IoT (via IoT Central), Machine Learning / AI, Azure Cognitive Services (APIs/SDKs that help developers build cognitive intelligence across Decisions, Vision, Speech, Language and Web-search)

“We will continue to partner with existing suppliers, emerging innovators and network equipment partners to share roadmaps and explore expanded opportunities to work together, including in the areas of radio access networks (RAN), next-generation core, virtualized services, orchestration and operations support system/business support system (OSS/BSS) modernization,” states Yousef Khalidi in this Microsoft post.

Acquisition of Metaswitch Networks (a provider of virtualised network software) and Affirmed Networks (a provider that sells virtualised, cloud-native mobile network solutions) shows further evidence of ambitions in the telco / cloud domain.

Like the partnerships described with AWS and Google above, Netcracker has also partnered with Microsoft, offering its Netcracker Digital BSS/OSS and Orchestration applications on Microsoft Azure. This article also describes that, “Netcracker is collaborating with Microsoft to integrate Azure Machine Learning (ML) and AI services with Netcracker’s Advanced Analytics to add intelligent contextual decisioning and recommendations to enable more personalized customer engagements.”

Meanwhile, Amdocs and Microsoft have been working on making ONAP available on the Azure platform. Nokia and Microsoft are partnering on “…cloud, Artificial Intelligence (AI) and Internet of Things (IoT), bringing together Microsoft cloud solutions and Nokia’s expertise in mission-critical networking.”

Microsoft Azure Edge Zones are offered through Azure, with select carriers and operators, or as private customer zones. They bring compute, storage, and service availability closer to the customer / device for low-latency + high-throughput use cases.

The recent announcement of an AT&T and Microsoft alliance (and more recently here where AT&T will run its 5G mobility network over the Azure cloud) as well as deals involving Telefónica (with Aura, its AI-powered digital assistant), SK Telecom (5G-based cloud gaming), Reliance Jio (cloud solutions), NTT (enterprise solution offerings), and Etisalat (future networks) show an increasing presence for Azure within the telco domain.

Netcracker has integrated its solution with Microsoft business applications (eg Office 365, Dynamics 365 and OneDrive), as other OSS/BSS providers undoubtedly have too.

Microsoft also has their own OSS/BSS offering in Dynamics 365 Field Service Management.

VMware Telco Cloud

telco.vmware.com 

VMware telco cloud architecture

VMware’s recently announced 5G Telco Cloud Portfolio has been designed to give network operators the platform to accelerate 5G and Edge implementation. Its key differentiator from the examples provided above is it allows operators to run containerised workloads across private, telco, edge and public clouds. This is seen as being an important feature allowing telcos to avoid cloud partner lock-in.

The press release above indicates that, “VMware is evolving its VMware vCloud NFV solution to Telco Cloud Infrastructure, providing CSPs a consistent and unified platform delivering consistent operations for both Virtual Network Functions (VNFs) and Cloud Native Network Functions (CNNFs) across telco networks. Telco Cloud Infrastructure is designed to optimize the delivery of network services with telco centric enhancements, supporting distributed cloud deployments…
Tightly integrated with Telco Cloud Infrastructure, VMware’s Telco Cloud Automation intelligently automates the end-to-end lifecycle management of network functions and services to simplify operations and accelerate service delivery while optimizing resource utilization. Telco Cloud Automation also now supports infrastructure and Containers-as-a-Service (CaaS) management automation to streamline workload placement and deliver optimal infrastructure resource allocation. It also significantly simplifies the 5G and telco edge network expansions through zero-touch-provisioning (ZTP) whenever capacity is required.

Red Hat

www.redhat.com/telco

Red Hat’s origins, of course, are in open-source tools such as Red Hat Linux. They’ve now evolved to become a leading provider of open-source solutions for enterprise, delivering Linux, cloud, container and Kubernetes technology. Red Hat was acquired by IBM in 2019.

Red Hat’s telco offerings are built upon the premise that service providers will use more open-source, multi-vendor solutions to underpin their OSS, BSS and networks of the future. Red Hat aims to offer open infrastructure to facilitate service provider initiatives such as NFV, 5G, OpenRAN and Edge Compute. This includes coordination of telco and IT infrastructure, but also the applications and data that are intertwined with them.

Red Hat’s telco proposition is supported by:

  • OpenShift – a cloud compute platform as a service (PaaS), built upon Docker containers (containerised applications) that are managed by Kubernetes. This is particularly relevant to support telco cloud models that provide virtualised network functions. It also helps to deliver the edge compute infrastructure that’s becoming synonymous with 5G
  • OpenStack – a set of components, mostly deployed as infrastructure as a service (IaaS) that help manage compute, storage and networks. Some of the components are shown in the diagram below sourced from redhat.com
    OpenStack Components (not complete)
  • Ansible Automation Platform – to automate network configuration, fault remediation, security updates, and more
  • Marketplace – to assist service providers in finding, buying and deploying certified, container-based software
  • Telco Ecosystem Program – that brings together enterprise and community partners to deliver integrated telco solutions. Partners include Affirmed, Altiostar, Atos, Cisco, Ericsson, Amdocs, MYCOM OSI, Zabbix, Metaswitch, Nokia, Juniper and more
  • Consulting – offering service resources that include Innovation Labs, training and consulting
  • And other solutions such as Ceph Storage, Cloud Suite, Quay, JBoss suite, Integration, Insights, Fuse and more.

Summary

CSPs (Communications Service Providers) find themselves in a catch-22 position with cloud providers. Their own OSS/BSS and those of their suppliers have an increasing reliance on cloud provider services and infrastructure. Due to economies of scale, efficiency of delivery, scalability and a long-tail of service offerings (from the cloud providers and their marketplaces), CSPs aren’t able to compete. Complexity of public cloud (security, scalability, performance, interoperability, etc) also make it a quandary for CSPs. It’s already a challenge (commercially and technically) to run the networks they do, but prohibitively difficult to expand coverage further to include public cloud. 

Yet, by investing heavily in cloud services, CSPs are funding further growth of said cloud providers, thus making CSPs less competitive, but more reliant, on cloud services. Telco architects are becoming ever more adept at leveraging the benefits of cloud. An example is being able to spin up apps without having to wait for massive infrastructure projects to be completed first, which has been a massive dependency (ie time delay) for many OSS/BSS projects.

In the distant past, CSPs had the killer apps, being voice and WAN data. These services supported the long-tail of business (eg salespeople from every industry in the world would make sales calls via telephony services) and customers were willing to pay a premium for these services.

The long-tail of business is now omni-channel, and the killer apps are content, experiences, data and the apps that support them. Being the killer apps, whoever supplies them also takes the premium and share-of-wallet. AWS, Google and Microsoft are supplying more of today’s killer apps (or the platforms that support them) than CSPs are.

The risk for CSPs is that cloud providers and over the top players will squeeze most of the profits from massive global investments in 5G. This is exacerbated if telco architects get their cloud architectures wrong and OPEX costs spiral out of control. Whether architectures are optimal or not, CSPs will fund much of the cloud infrastructure. But if CSPs don’t leverage cloud provider offerings, the infrastructure will cost even more, take longer to get to market and constrain them to local presence, leaving them at a competitive disadvantage with other CSPs. 

If I were a cloud provider, I’d be happy for CSPs to keep providing the local, physical, outside plant networks (however noting recent investments in local CSPs such as Amazon’s $2B stake in Bharti Airtel and Google’s $4.7 billion investment in Jio Platforms* not to mention Google Fiber and sub-sea fibre roll-outs such as this). It’s CAPEX intensive and needs a lot of human interaction to maintain / augment the widely distributed infrastructure. That means a lot is paid on non-effective time (ie the travel-time of techs travelling to site to fix problems, managing resources and/or coordinating repairs with property owners). Not only that, but there tends to be a lot of regulatory overhead managing local infrastructure / services as well as local knowledge / relationships. Governments want to ensure all their constituents have access to communications services at affordable prices. All the while, revenue per bit is continuing to drop, so merely shuffling bits around is a business model with declining profitability.

With declining profitability, operational efficiency improvements and cost reduction becomes even more important. OSS/BSS tools are vital for delivering improved productivity. But CSPs are faced with the challenge of transforming from legacy, monolithic OSS/BSS to more modern, nimble solutions. The more modular, flexible OSS/BSS of today and in future roadmaps are virtualised, microservice-based and designed for continuous delivery / DevOps. This is painting CSPs into a cloud-based future.

Like I said, a catch-22 for CSPs!

But another interesting strategy by Google is that its Anthos hybrid cloud platform will run multi-cloud workloads, including workloads on AWS and Microsoft Azure. Gartner predicts that >75% of midsize and large organisations will have adopted a multi-cloud and/or hybrid IT strategies by 2021 to prevent vendor lock-in. VMware (Dell) and Red Hat (IBM) are others creating multi-cloud / hybrid-cloud offerings. This gives the potential for CSPs to develop a near-global presence for virtualised telco functions. But will cloud providers get there before the telcos do?

For those of us supporting or delivering OSS/BSS, our future is in the clouds either way. It’s a rapidly evolving landscape, so watch this space.

 

* Note: Google is not the only significant investor in Jio:

Investor US$B Stake
Facebook 5.7 10%
Silver Lake Partners 1.43 2.08%
Mubadala 1.3 1.85%
Adia UAE Sovereign 0.8 1.16%
Saudi Arabia Sovereign 1.6 2.32%
TPG 0.64 0.93%
Catterton 0.27 0.39%
Intel 0.253 0.39%
Qualcomm 0.097 15.00%
Google 4.7 7.70%
TOTALS 16.79 42%

 

OSS sell money!

Huh? But they’re just cost centres aren’t they?
 
Nope, they sell financial outcomes – they reduce downtime, they turn on revenue, they improve productivity by coordinating the workforce, etc…
 
But they only “sell money” if they can help stakeholders clearly see the money! I mean “actually” see it, not “read between the lines” see it! (so many benefits of OSS are intangible, so we have to help make the financial benefits more obvious).
 
They don’t sell network performance metrics or orchestration plans or AI or any other tech chatter. They sell money in the form of turning on customers that pay to use comms services. They sell insurance policies (ie service reliability) that keep customers from churning.
 
Or to think of it another way, could you estimate (in a dollar amount) the consequences of not having the OSS/BSS? What would the cost to your organisation be?

OSS diamonds are forever (part 2)

Wednesday’s post discussed how OPEX is forever, just like the slogan for diamonds.
 
As discussed, some aspects of Operational Expenses are well known when kicking off a new OSS project (eg annual OSS license / support costs). Others can slip through the cracks – what I referred to as OPEX leakage (eg third-party software, ongoing maintenance of software customisations).
 
OPEX leakage might be an unfair phrase. If there’s a clear line of sight from the expenses to a profitable return, then it’s not leakage. If costs (of data, re-work, cloud services, applications, etc) are proliferating with no clear benefit, then the term “leakage” is probably fair.
 
I’ve seen examples of Agile and cloud implementation strategies where leakage has occurred. And even the supposedly “cheap” open-source strategies have led to surprises. OPEX leakage has caused project teams to scramble as their financial year progressed and budgets were unexpectedly being exceeded.
 
Oh, and one other observation to share that you may’ve seen examples of, particularly if you’ve worked on OSS in large organisations – Having OPEX incurred by one business unit but the benefit derived by different business units. This can cause significant problems for the people responsible for divisional budgets, even if it’s good for the business as a whole. 
 
Let me explain by example: An operations delivery team needs extralogging capability so they stand up a new open-source tool. They make customisations so that log data can be collected for all of their network types. All log data is then sent to the organisation’s cloud instance. The operations delivery team now owns lifecycle maintenance costs. However, the cost of cloud (compute and storage) and data lake licensing have now escalated but Operations doesn’t foot that bill. They’ve just handed that “forever” budgetary burden to another business unit.
 
The opposite can also be true. The costs of build and maintain might be borne by IT or ops, but the benefits in revenue or CX (customer experience) are gladly accepted by business-facing units.
 
Both types of project could give significant whole-of-company benefit. But the unit doing the funding will tend to choose projects that are less effective if it means their own business unit will derive benefit (especially if individual’s bonuses are tied to those results).
 
OSS can be powerful tools, giving and receiving benefit from many different business units. However, the more OPEX-centric OSS projects that we see today are introducing new challenges to get funded and then supported across their whole life-cycle.
 
PS. Just like diamonds bought at retail prices, there’s a risk that the financials won’t look so great a year after purchase. If that’s the case, you may have to seek justification on intangible benefits.  😉
 
PS2. Check out Robert’s insightful comment to the initial post, including the following question, “I wonder how many OSS procurements are justified on the basis of reducing the Opex only *of the current OSS*, rather than reducing the cost of achieving what the original OSS was created to do? The former is much easier to procure (but may have less benefit to the business). The latter is harder (more difficult analysis to do and change to manage, but payoff potentially much larger).”

Crossing the OSS chasm

Geoff Moore’s seminal book, “Crossing the Chasm,” described the psychological chasm between early buyers and the mainstream market.

Crossing the Chasm

Seth Godin cites Moore’s work, “Moore’s Crossing the Chasm helped marketers see that while innovation was the tool to reach the small group of early adopters and opinion leaders, it was insufficient to reach the masses. Because the masses don’t want something that’s new, they want something that works…

The lesson is simple:

– Early adopters are thrilled by the new. They seek innovation.

– Everyone else is wary of failure. They seek trust.”
 

I’d reason that almost all significant OSS buyer decisions fall into the “mainstream market” section in the diagram above.  Why? Well, an organisation might have the 15% of innovators / early-adopters conceptualising a new OSS project. However, sign-off of that project usually depends on a team of approvers / sponsors. Statistics suggest that 85% of the team is likely to exist in a mindset beyond the chasm and outweigh the 15%. 

The mainstream mindset is seeking something that works and something they can trust.

But OSS / digital transformation projects are hard to trust. They’re all complex and unique. They often fail to deliver on their promises. They’re rarely reliable or repeatable. They almost all require a leap of faith (and/or a burning platform) for the buyer’s team to proceed.

OSS sellers seek to differentiate from the 400+ other vendors (of course). How do they do this? Interestingly, by pitching their innovations and uniqueness mostly.

Do you see the gap here? The seller is pitching the left side of the chasm and the buyer cohort is on the right.

I wonder whether our infuriatingly lengthy sales cycles (often 12-18 months) could be reduced if only we could engineer our products and projects to be more mainstream, repeatable, reliable and trustworthy, whilst being less risky.

This is such a dilemma though. We desperately need to innovate, to take the industry beyond the chasm. Should we innovate by doing new stuff? Or should we do the old, important stuff in new and vastly improved ways? A bit of both??

Do we improve our products and transformations so that they can be used / performed by novices rather than designed for use by all the massive intellects that our industry seems to currently consist of?

 

 

 

 

Diamonds are Forever and so is OSS OPEX

Sourced from: www.couponraja.in

I sometimes wonder whether OPEX is underestimated when considering OSS investments, or at least some facets (sorry, awful pun there!) of it.

Cost-out (aka head-count reduction) seems to be the most prominent OSS business case justification lever. So that’s clearly not underestimated. And the move to cloud is also an OPEX play in most cases, so it’s front of mind during the procurement process too. I’m nought for two so far! Hopefully the next examples are a little more persuasive!

Large transformation projects tend to have a focus on the up-front cost of the project, rightly so. There’s also an awareness of ongoing license costs (usually 20-25% of OSS software list price per annum). Less apparent costs can be found in the exclusions / omissions. This is where third-party OPEX costs (eg database licenses, virtualisation, compute / storage, etc) can be (not) found.

That’s why you should definitely consider preparing a TCO (Total Cost of Ownership) model that includes CAPEX and OPEX that’s normalised across all options when making a buying decision.

But the more subtle OPEX leakage occurs through customisation. The more customisation from “off-the-shelf” capability, the greater the variation from baseline, the larger the ongoing costs of maintenance and upgrade. This is not just on proprietary / commercial software, but open-source products as well.

And choosing Agile almost implies ongoing customisation. One of the things about Agile is it keeps adding stuff (apps, data, functions, processes, code, etc) via OPEX. It’s stack-ranked, so it’s always the most important stuff (in theory). But because it’s incremental, it tends to be less closely scrutinised than during a CAPEX / procurement event. Unless carefully monitored, there’s a greater chance for OPEX leakage to occur.

And as we know about OPEX, like diamonds, they’re forever (ie the costs re-appear year after year).