DTA is all wrapped up for another year

We’ve just finished the third and final day at TM Forum’s Digital Transformation Asia (https://dta.tmforum.org and #tmfdigitalasia ). Wow, talk about a lot happening!!

After spending the previous two days focusing on the lecture series, it would’ve been remiss of me to not catch up with the vendors and Catalyst presentations that had been on display for all three days. So that was my main focus for day 3. Unfortunately, I probably missed seeing some really interesting presentations, although I did catch the tail-end of the panel discussion, “Zero-touch – Identifying the First Steps Toward Fully Automated NFV/SDN,” which was ably hosted by George Glass (along with NFV/SDN experts Tomohiro Otani and Ir. Rizaludin Kaspin ). From the small amount I did see, it left me wishing that I could’ve experienced the entire discussion.

But on with the Catalysts, which are one of the most exciting assets in TM Forum’s arsenal IMHO. They connect carriers (as project champions) with implementers to deliver rapid prototypes on some of the carriers’ most pressing needs. They deliver some seriously impressive results in a short time, often with implementers only being able to devote part of their working hours (or after-hours) to the Catalyst.

As reported here, the winning Catalysts are:

1. Outstanding Catalyst for Business Impact
Telco Cloud Orchestration Plus, Using Open APIs on IoT
Champion: China Mobile
Participants: BOCO Inter-Telecom, Huawei, Hewlett Packard Enterprise, Nokia

2. Outstanding Catalyst for Innovation
5G Pâtisserie
Champions: Globe Telecom, KDDI Research, Singtel
Patricipants: Neural Technologies, Infosys, Ericsson

3. Outstanding New Catalyst
Artificial Intelligence for IT Operations (AIOps)
Champions: China Telecom, China Unicom, China Mobile
Participants: BOCO Inter-Telecom, Huawei, Si-Tech

These were all undoubtedly worthy winners, reward for the significant effort that has already gone into them. Three other Catalysts that I particularly liked are:

  • Transcend Boundaries – which demonstrates the use of Augmented Reality for the field workforce in particular, as championed by Globe. Collectively we haven’t even scratched the surface of what’s possible in this space, so it was exciting to see the concept presented by this Catalyst
  • NaaS in Action – which is building upon Telstra’s exciting Network as a Service (NaaS) initiative; and
  • Telco Big Data Security and Privacy Management Framework – the China Mobile led Catalyst that is impressive for the number of customers that are already signed up and generating revenues for CT.

BTW. The full list of live Catalysts can be found here.

For those who missed this year’s event, I can only suggest that you mark it in your diaries for next year. The TM Forum team is already starting to plan out next year’s event, one that will surely be even bigger and better than the one I’ve been privileged to have attended this week.

Is OSS the future of OSS?

Don’t worry. The title of this post isn’t a typo, but I’ll get to that shortly.

I’ve just had an interesting day 2 at TM Forum’s Digital Transformation Asia (https://dta.tmforum.org and #tmfdigitalasia ). The quality of presentations was again quite high with further thought-provoking ideas!!

My favorite session for the day was a panel discussion entitled, “Is open-source the future of OSS/BSS?” Hence the title of today’s blog. Is OSS (open source software) the future of OSS?

Trevor Cheung of OpenROADS Community spoke about their framework for delivering transformation. One point he emphasised was that we’re so wrapped up in Customer Experience (CX), we often forget about Employee Experience. Put simply, if we don’t win the hearts and minds of the implementers, there’s never going to be a transformed experience for the customers to have.

Jurgen Hase of unlimit gave a number of really interesting perspectives, but the best is paraphrased as follows, “The S in IoT stands for security… Wait, what? There is no S in IoT??”

Next was Angelia Ooi of TIME. Angelia provided 8 really useful tips on digital transformation via a presentation pack that is easily the most succinct and polished of all those I’ve seen at DTA so far.

Joddy Hernady of Telkom Indonesia provided some of the economics of becoming a digital telco, which provided an interesting perspective on the benefits of achieving digital transformation.

But finally, and it was the last presentation of the day that was most thought-provoking. Is open source the future of OSS/BSS?
Unfortunately I missed almost all of Catherine Michel’s opening gambit, but I believe the CTO of Sigma Systems made the key point that open source projects such as Mongo DB should really only be considered once they’d reached a level of maturity, ongoing development and support that approaches the large ISVs (Independent Software Vendors) such as Sigma Systems. She also highlighted the multi-layered challenges around licensing / rights.
Gnanapriya Chidambaranathan of Infosys contended that there is a wealth of open source projects that can be leveraged, curated and supported by integrators such as Infosys. She posed that open source adoption is a key to innovation.
Venura Mendis of Apigate provided the perspective of an open source software provider. He highlighted the challenge he faces in dealing with traditional carrier procurement teams, particularly in their ambition of reaching comparative TCO (Total Cost of Ownership) models.
Guy Lupo of Telstra provided a number of different and interesting perspectives, as he regularly does, this time on a carrier deciding between ISV, open source products and going down the path (rabbit hole?) of open sourcing their own developments. Guy’s perspectives were really pertinent as he’s currently utilising all of these options in his NaaS (Network as a Service) program at Telstra.

Finally, a few thoughts from me on the topic of OSS as the future of OSS.

1. One of the biggest challenges facing the future of OSS is fragmentation. The PAOSS vendor list has over 200 records (and I’ll be doing a major update again shortly that will add hundreds of additional vendors). This means the available skills pool is diluted with a lot of functionality duplication. It also means it becomes really challenging for customers to choose the right product for their needs (although we could claim that this is a good thing for PAOSS as we often assist customers with this challenge). The proliferation of open source projects that deliver OSS/BSS functionality further fragments and dilutes

2. We’re seeing a trend away from the behemoth software stacks of the past for a variety of reasons, but could be summed up as the laws of physics preventing us from making a large-scale OSS pivots. The more modular OSS appear to be more nimble. This plays into the hands of niche open source offerings. It appears contra to the massive-scale open source efforts of ONAP, which interestingly, the above mentioned panelists also held doubts over ONAP’s ability to succeed. I should note that they, like me, were also enthusiastic about facets of ONAP such as the collaboration, initiative taken, etc.

3. I still believe there is the potential to build an open-source OSS core that then allows collaboration and plug-ins to be developed, thus better leveraging the long tail of innovation from the available skills pool. Today’s panelists did throw something of a spanner in these works though by pointing out the layered licensing challenge with open source. It’s quite common for open source projects to leverage open source projects, which in turn leverage open source projects. Guy in particular highlighted just how big a problem it has been for Telstra’s procurement team to trace out all the open source threads.

The biggest OSS loser

You are so much more likely to put effort into something when you know whether it will pay off and what the gains will be. Not knowing how things will turn out undermines your motivation and makes you delay taking action.”
Dr Theo Tsaousides
in his book, Brainblocks.

Have you seen the reality TV show, “The Biggest Loser?” I rarely watch TV, but have noticed that it’s been a runaway hit in the ratings here in Australia (and overseas apparently). Why has it been so successful and what does it have to do with OSS?

Well, according to Dr Tsaousides, the success of the show comes down to the obvious body-shape / fitness transformations each of the contestants makes over each season of the show. But more specifically, “You need to watch only one season from beginning to end and you will start craving to be a contestant on the show, regardless of your current weight… Seeing the people’s amazing transformation over a few months is a much more convincing way to start working out and eating well than being told by your doctor that you need to lose weight and about the cardiovascular advantages of exercise. Forecasting a positive outcome, especially when dealing with something new and unfamiliar, leads to action.”

Can you see how this might be a useful technique when planning an OSS transformation?

Change management is always a challenging task on any large OSS transformation. It’s always best to have the entire OSS user population involved in the change, but that’s not always feasible for large groups of users.

It’s one of the reasons I’m always a big advocate for getting a baseline, sandpit version of off-the-shelf OSS stood up and available for the user population to start interacting with. This is particularly helpful if the sandpit is perceptibly better than the current one.

To paraphrase, “Forecasting a positive outcome (via the OSS sandpit), especially when dealing with something new and unfamiliar (the future state after OSS transformation), leads to action (more excitement, engagement and less pushback from the user population during the course of the transformation).”

Do you think the biggest loser technique could work on your next OSS transformation?

Presence vs omni-presence and the green button of OSS design

In OSS there are some tasks that require availability (the green button on communicator). The Network Operations Centre (NOC) is one. But does it require on-site presence in the NOC?

An earlier post showed how wrong I was about collaboration rooms. It seems that ticket flicking (and perhaps communication tools like slack) is the preferred model. If this is the preferred model, then perhaps there is no need for a NOC… perhaps only a DR NOC (Disaster Recover NOC).

Truth is, there are hardly any good reasons to know if someone’s available or away at any given moment. If you truly need something from someone, ask them. If they respond, then you have what you needed. If they don’t, it’s not because they’re ignoring you – it’s because they’re busy. Respect that! Assume people are focused on their own work.
Are there exceptions? Of course. It might be good to know who’s around in a true emergency, but 1% occasions like that shouldn’t drive policy 99% of the time. 
Jason Fried on Signal v Noise

Customer service needs availability. But with a multitude of channels (for customers) and collaboration tools (for staff*), it decreasingly needs presence (except in retail outlets perhaps). You could easily argue that contact centres, online chat operators, etc don’t require presence, just availability.

The one area where I’m considering the paradox of presence is in OSS design / architecture. There are often many facets of a design that require multiple SMEs – OSS application, security, database, workflow, user-experience design, operations, IT, cloud etc.

When we get many clever SMEs in the one room, they often have so many ideas and so much expertise that the design process resembles an endless loop. Presence seems to inspire omnipresence (the need to show expertise across all facets of the design). Sometimes we achieve a lot in these design workshops. Sometimes we go around in circles almost entirely because of the cleverness of our experts. They come up with so many good ideas we end up in paralysis by analysis.

The idea I’m toying with is how to use the divide and conquer theory – being able to carve up areas of responsibility and demarcation points to ensure each expert focuses on their area of responsibility. Having one expert come up with their best model within their black box of responsibility and connecting their black box with adjacent demarcation points. The benefits are also the detriments. The true double-edged sword. The benefits are having one true expert work through the options within the black box. The detriments are having only one expert work through the options within the black box.

There are some past projects that I wished I’d tried to inspire the divide and conquer approach in hindsight. In others, the collaboration model has worked extremely well.

But to get back to presence, I wonder whether thrashing up front to define black boxes and demarcation points then allows the experts to do their thing remotely and become less inclined to analyse and opine on everyone else’s areas of expertise.

* I use the term staff to represent anyone representing the organisation (staff, contractor, consultant, freelancer, etc)

Intent to simplify our OSS

The left-hand panel of the triptych below shows the current state of interactions with most OSS. There are hundreds of variants inbound via external sources (ie multi-channel) and even internal sources (eg different service types). Similarly, there are dozens of networks (and downstream systems), each with different interface models. Each needs different formatting and integration costs escalate.
Intent model OSS

The intent model of network provisioning standardises the network interface, drastically simplifying the task of the OSS and the variants required for it to handle. This becomes particularly relevant in a world of NFVs, where it doesn’t matter which vendor’s device type (router say) can be handled via a single command intent rather than having separate interfaces to each different vendor’s device / EMS northbound interface. The unique aspects of each vendor’s implementation are abstracted from the OSS.

The next step would be in standardising the interface / data model upstream of the OSS. That’s a more challenging task!!

What if the OSS solution lies in its connections?

Imagine for a moment that you’re sitting in front of a pristine chess board, awaiting the opportunity to make your first move. All of the pieces have been exquisitely carved from stone, polished to a sheen. The rules of the game have been established for centuries, so you know exactly which piece is able to move in which sequences. Time to make the opening move.

You’ve studied the games of the masters who have preceded you and have planned your opening gambit, the procession of moves that will hopefully take you into a match-winning position. Due to your skills with modern automations, you’ve connected some of the chess pieces with delicate strings to implement your opening gambit with precision.

Unfortunately, after the first few moves, your strings are starting to pull the pieces out of position. Your opponent has countered well and you’re having to modify your initial plans. You introduce some additional pulleys and springs to help retain the rightful position of your pieces on the board and cope with unexpected changes in strategy. The automations are becoming ever more complex, taking more time to plan and implement than the actual next move.

The board is starting to devolve into unmanageable chaos.

Does this sound like the analogy of a modern OSS? It’s what I refer to as the chessboard analogy.

We’ve been at this OSS game for long enough to already have an understanding of all of the main pieces. TM Forum’s TAM provides this definition as a useful guide. The pieces are modular, elegant and quite well understood by its many players. The rules of the game haven’t really changed much. The main use cases of an OSS from decades ago (ie assure, fulfil, plan, build, etc) probably don’t differ significantly from those of today. This
“should” set the foundations for interchangeability of applications.

We see programs of work like ONAP, where millions of lines of code are being developed to re-write the rules of the game. I’m a big advocate of many of the principles of ONAP, but I’m still not sure that such a massive re-write is what’s needed.

It’s not so much in the components of our OSS as in the connections between them where things tend to go awry.

The foundation of all brilliance is seeing connections when no one else does.”
Richard Parkinson
.

This article distills ONAP from its answers back to the core questions. What if instead of seeking an entirely-new architectural stack, we focused on solving the core questions and the chessboard problem – the problem of connections?

Perhaps the answer to the connection problem lies in the interchangeable small grid OSS model discussed in yesterday’s article on planned OSS obsolescence.
But it probably also incorporates what ONAP calls, “real-time, policy-driven orchestration and automation,” to replace pre-defined processes. I wonder instead whether state-based transitions, being guided by intent/policy rules and feedback loops (ie learning systems) might hold the key. An evolving and learning solution that shares similarities with the electrical pathways in our brain, which strengthen the more they’re used and diminish if no longer used.

The future of work and its impact on OSS

Many years ago, I worked on a seriously big OSS transformation for one of the region’s biggest telcos. Everything was big on the project, the investment, the resources, the documentation. Everything except the outcomes. There was so much inefficiency that I often spoke about making one day of progress for every ten on site. Meetings, bureaucracy, impossible approval cycles, customer re-organisations, over-analysis, etc all added up to stagnation.

This contrasted so much with some of the amazing small teams I’ve worked alongside. Teams that worked cohesively, cleverly and just got stuff done with almost no resources. It’s one of the reasons I feel that the future of work, even for the very large organisations, will be via small teams. Outsourced to small, efficient teams / organisations. The gig economy, and the proliferation of tools that support it, make it an obvious approach to take, especially for very large organisations to leverage. Proof of work technologies, such as those building upon the discovery of blockchain, will provide further impetus to use smaller teams of experts.

Experts like a friend and colleague of mine who once built an alarm management tool in a weekend, by himself. It also happened to be more sophisticated than his employer’s existing tool that had taken years of combined developer effort by a larger team.

Maybe I’ll be proven wrong, but I see the transition to this model of work as being inevitable. The question I have is how to make our OSS more accommodating of this work model. Behemoth OSS stacks won’t. Highly modular OSS made up of many smaller components probably will, as long as they don’t succumb to the OSS chessboard analogy. The pulleys and strings will make it impossible for small, interchangable teams to decipher and manage.

A small-grid OSS model is the one I’d be backing in.

OSS – like a duck on a pond

Let’s start with a basic question. “What does an OSS need to do?”

The basic answer is, “make operations easier.”

The real answer(s) is so much more nuanced than that of course. The term easier can also encapsulate other words such as faster, more accurate, more repeatable, cheaper, etc.

Designing, building, operating and maintaining a sizable network is extremely challenging, despite network operators around the world, and the vendors that supply to them, employing some of the best and brightest. So we design OSS and related tools / processes to make operations easier.

Yet I sometimes wonder whether we achieve that aim – to make operations easier. Seems to me that we tend to focus more on just replicating functions at a higher layer in the management stack. That is, moving the function to the OSS rather than EMS/NMS, without really making it much easier operationally.

Let’s start at the user interface (UI). How often are they intuitive enough for an experienced network operator to start doing tasks with negligible OSS expert guidance?
Let’s look at deployments. How often are the projects low on effort, risk, cost and complexity?
Let’s look at flexibility (ie in-flight modifications or transformations). How often do we actually deliver flexibility to our customers through our OSS. To ask the same as above, how often are our changes low on effort, risk, cost and complexity?

As a small step towards providing an answer, I wonder whether it’s a case of making the hard things look easy and the easy things look hard.

We want to make the really hard operational things much easier to do within an OSS because that’s the primary purpose of an OSS. That’s the example of a duck on a pond. The OSS is gliding along effortlessly across the top of the water, but under the water it is paddling furiously.

Conversely, we want to make the really easy* operational things look hard to do within an OSS so that we’re not constantly being asked to build functionality / complexity into our OSS that doesn’t warrant being there. It diffuses the intent of the OSS. Just because we can, doesn’t mean we should.

Build an OSS and they will come… or sometimes not

Build it and they will come.

This is not always true for OSS. Let me recount a few examples.

The project team is disconnected from the users – The team that’s building the OSS in parallel to existing operations doesn’t (or isn’t able to) engage with the end users of the OSS. Once it comes time for cut-over, the end users want to stick with what they know and don’t use the shiny new OSS. From painful experience I can attest that stakeholder management is under-utilised on large OSS projects.

Turf wars – Different groups within a customer are unable to gain consensus on the solution. For example, the operational design team gains the budget to build an OSS but the network assurance team doesn’t endorse this decision. The assurance team then decides not to endorse or support the OSS that is designed and built by the design team. I’ve seen an OSS worth tens of millions of dollars turned off less than 2 years after handover because of turf wars. Stakeholder management again, although this could be easier said than done in this situation.

It sounded like a good idea at the time – The very clever OSS solution team keeps coming up with great enhancements that don’t get used, for whatever reason (eg non fit-for-purpose, lack of awareness of its existence by users, lack of training, etc). I’ve seen a customer that introduced over 500 customisations to an off-the-shelf solution, yet hundreds of those customisations hadn’t been touched by users within a full year prior to doing a utilisation analysis. That’s right, not even used once in the preceding 12 months. Some made sense because they were once-off tools (eg custom migration activities), but many didn’t.

The new OSS is a scary beast – The new solution might be perfect for what the customer has requested in terms of functionality. But if the solution differs greatly from what the operators are used to, it can be too intimidating to be used. A two-week classroom-based training course at the end of an OSS build doesn’t provide sufficient learning to take up all the nuances of the new system like the operators have developed with the old solution. Each significant new OSS needs an apprenticeship, not just a short-course.

It’s obsolete before it’s finishedOSS work in an environment of rapid change – networks, IT infrastructure, organisation models, processes, product offerings, regulatory shifts, disruptive innovation, etc, etc. The longer an OSS takes to implement, the greater the likelihood of obsolescence. All the more reason for designing for incremental delivery of business value rather than big-bang delivery.

What other examples have you experienced where an OSS has been built, but the users haven’t come?

Falsely rewarding based on OSS existence rather than excellence

There’s a common belief that most jobs see people rewarded for presence rather than performance. That is, they’re encouraged to be on site from 9am to 5pm rather than being given free reign over their work schedules as long as key outcomes are met / exceeded.

In OSS vendor / product selection there’s a similar concept. Contracts are often awarded based on existence rather than excellence. When evaluating a product, if it’s able to do a majority of the functions in the long list of requirements then the box is ticked.

However, this doesn’t take into account that there are usually only a very small number of functions that any given customer’s OSS needs to perform at a very high level of efficiency. All the others are effectively just nice to have. That’s the 80/20 rule at work.

When guiding a customer through their vendor selections, I always take them through an exercise to identify the use-cases / functions that really matter. Then we ensure that the demos or proofs of concept focus closely on how excellent the OSS is at those most important factors.

OSS automations – just because we can, doesn’t mean we should

Automation is about using machines / algorithms to respond faster than humans can, or more efficiently than humans can, or more accurately than humans can… but only if the outcomes justify the costs. When it comes to automations, it’s a case of, “just because we can, doesn’t mean we should.”

The more complex the decision tree you’re trying to automate, the higher the costs and therefore the harder it becomes to cost-justify. So the first step in any automation is taking a lateral thinking approach to simplifying the decision tree.

This recent post highlighted a graph from Nokia’s Bell Labs and the financial dependency that network slicing has on operational automation:
Nokia Network Slicing

Let’s use the Toyota Five Whys technique to work our way through the implications of this:

Statement 0: As CSPs, we need to drastically reduce complexity in the processes / decision-trees across our whole organisation.

Why 1? So that we can apply significant levels of automation

Why 2? So that we can apply technologies / techniques such as network slicing or virtualisation that are cost-justifiable

Why 3? So that we can offer differentiated, premium services

Why 4? So that our offerings don’t become commodities

Why 5? So that we retain corporate profitability to return to shareholders and/or invest in further interesting projects

I love that we’re looking to all number of automation technologies / techniques to apply to our OSS. However, we’re bypassing the all-important statement 0. We’re starting at Why 1 and partially missing the cost-justifiable part of Why 2. If our automation projects don’t prove cost-justifiable, then we never get the chance to reach whys 3, 4 and 5.

OSS implementation, but without the dependencies

One of the challenges with getting a new OSS or OSS transformation project completed can be the large number of dependencies that can cause momentum gridlock. If you’re looking to deliver business value in one big-bang, which is a really common approach to delivering OSS projects, then you end up juggling many different activities and hoping they all align at the right times.

I’ve noticed that the vendors tend to design their delivery schedules around big-bang / waterfall approaches like below.
Big-bang OSS delivery

Many vendors will even assure you that this is their standard practice and are hesitant to consider changes to their “best practice” delivery scheduling. Having been involved in many of these types of deliveries in the past, on both vendor and customer side, I can assure you that they rarely work well.

Generally speaking, the gridlocks occur on the customer-side, but the result is detrimental to customer and vendor alike. Hold-ups mean inefficient allocation of resources as well as the resultant cost / time over-runs.

The alternative is to apply a bit more lateral thinking to how you break down the work into smaller chunks. The lateral thinking work breakdown aims are two-fold:

  1. How to break up the work so that it best avoids dependencies; whilst also
  2. Delivering some sort of value to the customer

There are many dependencies on a typical OSS project – hardware, procurement, IT infrastructure, network connectivity, security, approvals, integrations, licensing, resource availability, data quality and many more. However, each different customer, their org chart and project has its own unique mix of dependencies, so I don’t subscribe to the “best practice” argument to project delivery.

The diagram below shows an example of an alternate breakdown. The business value chunks that are delivered might be tiny in some cases, but at least momentum can be demonstrated. Rather than having a mass of entwined dependencies, you can isolate and minimise dependencies for that sliver of business value. When the dependency/ies has cleared, you can jump straight onto the next activity from an existing build-state rather than having to align all the activities to land in perfect precision.
Incremental OSS work breakdown

OSS project stalled? Cancel it

When a project appears to be in limbo, in a permanent holding pattern, where sunk costs meet opportunity costs, where no one can figure out what to do…

Cancel it.
Cancel it with a week’s notice.

One of two things will happen:
A. A surge of support and innovation will arrive, and it won’t be stuck any more.
B. You’ll follow through and cancel it, and you won’t be stuck any more.

It costs focus and momentum to carry around the stalled. Let it go.”
Seth Godin on his blog here.

OSS projects have a tendency to get so big and complex and with so many dependencies that they can stagnate. When projects stagnate, we have a tendency of treating them with contempt or cynicism don’t we? We treat them this way even when we’re involved, so you know that outsiders are treating them with even more contempt and cynicism.

So Seth’s concept is an interesting one. I haven’t tried his technique before.

Have you? Did it achieve your desired outcomes?
Did it rally the troops? Did it clear the way for assignment of resources onto better projects, Darwinian-style? Or did it just throw away the last vestiges of momentum and all sunk costs?

OSS designed as a bundle, or bundled after?

Over the years I’m sure you’ve seen many different OSS demonstrations. You’ve probably also seen presentations by vendors / integrators that have shown multiple different products from their suite.

How integrated have they appeared to you?

  1. Have they seemed tightly integrated, as if carved from a single piece of stone?
  2. Or have they seemed loosely integrated, a series of obviously different stones joined together with some mortar?
  3. Or perhaps even barely associated, a series of completely different objects (possibly through product acquisition) branded under a common marketing name?

There are different pros and cons with each approach. Tight integration possibly suits a greenfields OSS. Looser integration perhaps better suits carve-off for best-of-breed customer architecture models.

I don’t know about you, but I always prefer to be given the impression that an attempt has been made to ensure consistency in the bundling. Consistency of user-interface, workflow, data modelling/presentation, reports, etc. With modern presentation layers, database technologies and the availability of UX / CX expertise, this should be less of a hurdle than it has been in the past.

If ONAP is the answer, what are the questions?

ONAP provides a comprehensive platform for real-time, policy-driven orchestration and automation of physical and virtual network functions that will enable software, network, IT and cloud providers and developers to rapidly automate new services and support complete lifecycle management.
By unifying member resources, ONAP is accelerating the development of a vibrant ecosystem around a globally shared architecture and implementation for network automation–with an open standards focus–faster than any one product could on its own
.”
Part of the ONAP charter from onap.org.

The ONAP project is gaining attention in service provider circles. The Steering Committee of the ONAP project hints at the types of organisations investing in the project. The statement above summarises the mission of this important project. You can bet that the mission has been carefully crafted. As such, one can assume that it represents what these important stakeholders jointly agree to be the future needs of their OSS.

I find it interesting that there are quite a few technical terms (eg policy-driven orchestration) in the mission statement, terms that tend to pre-empt the solution. However, I don’t feel that pre-emptive technical solutions are the real mission, so I’m going to try to reverse-engineer the statement into business needs. Hopefully the business needs (the “why? why? why?” column below) articulates a set of questions / needs that all OSS can work to, as opposed to replicating the technical approach that underpins ONAP.

Phrase Interpretation Why? Why? Why?
real-time The ability to make instantaneous decisions Why1: To adapt to changing conditions
Why2: To take advantage of fleeting opportunities or resolve threats
Why 3: To optimise key business metrics such as financials
Why 4: As CSPs are under increasing pressure from shareholders to deliver on key metrics
policy-driven orchestration To use policies to increase the repeatability of key operational processes Why 1: Repeatability provides the opportunity to improve efficiency, quality and performance
Why 2: Allows an operator to service more customers at less expense
Why 3: Improves corporate profitability and customer perceptions
Why 4: As CSPs are under increasing pressure from shareholders to deliver on key metrics
policy-driven automation To use policies to increase the amount of automation that can be applied to key operational processes Why 1: Automated processes provide the opportunity to improve efficiency, quality and performance
Why 2: Allows an operator to service more customers at less expense
Why 3: Improves corporate profitability and customer perceptions
physical and virtual network functions Our networks will continue to consist of physical devices, but we will increasingly introduce virtualised functionality Why 1: Physical devices will continue to exist into the foreseeable future but virtualisation represents an exciting approach into the future
Why 2: Virtual entities are easier to activate and manage (assuming sufficient capacity exists)
Why 3: Physical equipment supply, build, deploy and test cycles are much longer and labour intensive
Why 4: Virtual assets are more flexible, faster and cheaper to commission
Why 5: Customer services can be turned up faster and cheaper
software, network, IT and cloud providers and developers With this increase in virtualisation, we find an increasingly large and diverse array of suppliers contributing to our value-chain. These suppliers contribute via software, network equipment, IT functions and cloud resources Why 1: CSPs can access innovation and efficiency occurring outside their own organisation
Why 2: CSPs can leverage the opportunities those innovations provide
Why 3: CSPs can deliver more attractive offers to customers
Why 4: Key metrics such as profitability and customer satisfaction are enhanced
rapidly automate new services We want the flexibility to introduce new products and services far faster than we do today Why 1: CSPs can deliver more attractive offers to customers faster than competitors
Why 2: Key metrics such as market share, profitability and customer satisfaction are enhanced as well as improved cashflow
support complete lifecycle management The components that make up our value-chain are changing and evolving so quickly that we need to cope with these changes without impacting customers across any of their interactions with their service Why 1: Customer satisfaction is a key metric and a customer’s experience spans the entire lifecyle of their service.
Why 2: CSPs don’t want customers to churn to competitors
Why 3: Key metrics such as market share, profitability and customer satisfaction are enhanced
unifying member resources To reduce the amount of duplicated and under-synchronised development currently being done by the member bodies of ONAP Why 1: Collaboration and sharing reduces the effort each member body must dedicate to their OSS
Why 2: A reduced resource pool is required
Why 3: Costs can be reduced whilst still achieving a required level of outcome from OSS
vibrant ecosystem To increase the level of supplier interchangability Why 1: To reduce dependence on any supplier/s
Why 2: To improve competition between suppliers
Why 3: Lower prices, greater choice and greater innovation tend to flourish in competitive environments
Why 4: CSPs, as customers of the suppliers, benefit
globally shared architecture To make networks, services and support systems easier to interconnect across the global communications network Why 1: Collaboration on common standards reduces the integration effort between each member at points of interconnect
Why 2: A reduced resource pool is required
Why 3: Costs can be reduced whilst still achieving interconnection benefits

As indicated in earlier posts, ONAP is an exciting initiative for the CSP industry for a number of reasons. My fear for ONAP is that it becomes such a behemoth of technical complexity that it becomes too unwieldy for use by any of the member bodies. I use the analogy of ATM versus Ethernet here, where ONAP is equivalent to ATM in power and complexity. The question is whether there’s an Ethernet answer to the whys that ONAP is trying to solve.

I’d love to hear your thoughts.

(BTW. I’m not saying that the technologies the ONAP team is investigating are the wrong ones. Far from it. I just find it interesting that the mission is starting with a technical direction in mind. I see parallels with the OSS radar analogy.)

Where are the reliability hotspots in your OSS?

As you already know, there are two categories of downtime – unplanned (eg failures) and planned (eg upgrades / maintenance).

Planned downtime sounds a lot nicer (for operators) but the reality is that you could call both types “incidents” – they both impact (or potentially impact) the customer. We sometimes underestimate that fact.

Today’s question is whether you’re able to identify where the hotspots are in your OSS suite when you combine both types of downtime. Can you tell which outages are service-impacting?

In a round-about way, I’m asking whether you already have a dashboard that monitors uptime of all the components (eg applications, probes, middleware, infra, etc) that make up your complete OSS / BSS estate? If you do, does it tell you what you anecdotally know already, or are there sometimes surprises?

Does the data give you the evidence you need to negotiate with the implementers of problematic components (eg patch cadence, the need for reliability fixes, streamlining the patch process, reduction in customisations, etc)? Does it give you reason to make architectural changes (eg webscaling)?

Persona mapping for OSS PoCs

When selecting new applications for an OSS or to augment an existing OSS, it always makes sense to me to run a Proof of Concept. But what do we want to demonstrate in that PoC? For me, we want to run demonstrations of the factors (eg features, use-cases, processes, etc) that justify the investment.

A simple exercise you can use is to identify the personas / roles that interact with the OSS. This could include personas such as NOC operator, strategic planner, network engineer, order entry, field ops, data / analytics, application administrator, etc. The actual personas will differ within each organisation of course.

For each of those personas, we can identify and interview an individual that represents that persona.

Interview questions include:

  1. What are the key responsibilities of your role
  2. What is the most important goal / KPI for your role
  3. How does this OSS (or proposed OSS) support you meeting this goal
  4. Describe the single most important process / function that you perform using the OSS
  5. Why is it so important
  6. How often do you perform this process / function
  7. Please provide a short list of other important processes / functions you perform with this OSS

We can then build this into a matrix and seek to prioritise into a set of use-cases. Based on time and cost constraints, we can then build the top-n of those use-cases into implementation scenarios for the PoC.

OSS operationalisation at scale

We had a highly flexible network design team at a previous company. Not because we wanted to necessarily, but because we were forced to by the client’s allocation of work.

Our team was largely based on casual workers because there was little to predict whether we needed 2 designers or 50 in any given week. The workload being assigned by the client was incredibly lumpy.

But we were lucky. We only had design work. The lumpiness in design effort flowed down through the work stack into construction, test and deployment teams. The constructors had millions of dollars of equipment that they needed to mobilise and demobilise as the work ebbed and flowed. Unfortunately for the constructors, they’d prepared their rate cards on the assumption of a fairly consistent level of work coming through (it was a very big project).

This lumpiness didn’t work out for anyone in the delivery pipeline, the client included. It was actually quite instrumental in a few of the constructors going into liquidation. The client struggled to meet roll-out targets.

The allocation of work was being made via the client’s B/OSS stack. The B/OSS teams were blissfully unaware of the downstream impact of their sporadic allocation of designs. Towards the end of the project, they were starting to get more consistent and delivery teams started to get into more of a rhythm… just as the network was coming to the end of build.

As OSS builders, we sometimes get so wrapped up in delivering functionality that we can forget that one of the key requirements of an OSS is to operationalise at scale. In addition to UI / CX design, this might be something as simple as smoothing the effort allocation for work under our OSS‘s management.

Help needed: IoT / OSS cross-over use cases

Hi PAOSS community.

I’d like to call in a favour today if I may. I’m on the hunt for any existing use-cases and / or project sites that have integrated a significant sensor network into their OSS and existing operational processes.

That includes a strategy for handling IoT-scale integration of data collection, event / alarm processing, device management, data contextualization, data analytics, end-to-end security and applications management / enablement within existing OSS tools.

I’m looking for examples where an OSS had previously managed thousands of (network) devices and is now managing hundreds of thousands of (IoT) devices. Not necessarily IoT devices of customers as services but within an operator’s own network.

Obviously that’s an unprecedented change in scale in traditional OSS terms, but will be commonplace if our OSS are to play a part in the management of large sensor networks in the future.

There’s an element of mutual exclusivity between what an IoT management platform and OSS needs to do, but there are also some similarities. I’d love to speak with anyone who has actually bridged the gap.

What OSS environments do you need?

When we’re planning a new OSS, we tend to be focused on the production (PROD) environment. After all, that’s where it’s primary purpose is served, to operationalise a network asset. That is where the majority of an OSS‘s value gets created.

But we also need some (roughly) equivalent environments for separate purposes. We’ll describe some of those environments below.

By default, vendors will tend to only offer licensing for a small number of database instances – usually just PROD and a development / test environment (DEV/TEST). You may not envisage that you will need more than this, but you might want to negotiate multiple / unlimited instances just in case. If nothing else, it’s worth bringing to the negotiation table even if it gets shot down because budgets are tight and / or vendor pricing is inflexible relating to extra environments.

Examples where multiple instances may be required include:

  1. Production (PROD) – as indicated above, that’s where the live network gets managed. User access and controls need to be tight here to prevent catastrophic events from happening to the OSS and/or network
  2. Disaster Recovery (DR) – depending on your high-availability (HA) model (eg cold standby, primary / redundant, active / active), you may require a DR or backup environment
  3. Sandpit (DEV / TEST) – these environments are essential for OSS operators to be able to prototype and learn freely without the risk of causing damage to production environments. There may need to be multiple versions of this environment depending on how reflective of PROD they need to be and how viable it is to take refresh / updates from PROD (aka PROD cuts). Sometimes also known as non-PROD (NP)
  4. Regression testing (REG TEST) – regression testing requires a baseline data set to continually test and compare against, flagging any variations / problems that have arisen from any change within the OSS or networks (eg new releases). This implies a need for data and applications to be shielded from the constant change occurring on other types of environments (eg DEV / TEST). In situations where testing transforms data (eg activation processes), REG TEST needs to have the ability to roll-back to the previous baseline state
  5. Training (TRAIN) – your training environments may need to be established with a repeatable set of training scenarios that also need to be re-set after each training session. This should also be separated from the constant change occurring on dev/test environments. However, due to a shortage of environments, and the relative rarity of training needed at some customers, TRAIN often ends up as another DEV or TEST environment
  6. Production Support (PROD-SUP) – this type of environment is used to prototype patches, releases or defect fixes (for defects on the PROD environment) prior to release into PROD. PROD-SUP might also be used for stress and volume testing, or SVT may require its own environment
  7. Data Migration (DATA MIG) – At times, data creation and loading needs to be prototype in a non-PROD environment. Sometimes this can be done in PROD-SUP or even a DEV / TEST environment. On other occasions it needs its own dedicated environment so as to not interrupt BAU (business as usual) activities on those other environments
  8. System Integration Testing (SIT)OSS integrate with many other systems and often require dedicated integration testing environments

Am I forgetting any? What other environments do you find to be essential on your OSS?