OSS Sandpit – 5G Network Inventory Prototype

5G networks seem to be the big investment trend in telco at the moment. It comes with a lot of tech innovation such as network slicing and an increased use of virtualised network functions (VNFs). This article provides an example of building 5G Network components into the inventory module of our Personal OSS Sandpit Project.

This prototype build includes components such as:

  • Hosting infrastructure
  • NFVI / VIM (NFV Infrastructure and Virtualised Infrastructure Management)
  • A 5GCN (5G Core Network)
  • An IMS (IP Multimedia Subsystem)
  • An RIC (RAN Intelligent Controller)
  • Virtualised Network Functions (AUSF, AMF, NRF, CU, DU, etc – a more extensive list of examples is provided later in this article)
  • Mobile Edge Compute (MEC)
  • MEC Applications like gaming servers, CDN (Content Delivery Networks)
  • Radio Access Network (RAN) and Remote Radio Units (RRU)
  • Outside Plant for fibre fronthaul and backhaul
  • Patching between physical infrastructure
  • End to end circuits between DN (Data Network), IMS, 5GCN, gNodeB, RRU
  • Logical Modelling of 5G Reference Points

Our prototype (a Standalone 5G model) is summarised in the diagram below:

We describe this via the following use-cases:

  • Building Reference Data like data hierarchies, device types, connectivity types, containment, device layouts, templates, flexible data models, etc
  • Creating Device Instances including rack views and the virtualised layers within them
  • Creating Physical Connections between devices
  • Creating Logical Connections between devices
  • Creating Network Slices in the form of services
  • Performing Service Impact Analysis (SIA)

Reference Data

Starting off with the data hierarchy, we had to develop some new building blocks (data classes) to support the virtualisation used in 5G networks. This included some new network slice types, virtualisation concepts and various other things:

In our prototype, we’ve developed a custom containment model as follows:

  • Country
    • Site
      • System (Network Domain)
        • Rack
          • Hosting
            • NFVI / VIM
              • VNF-Groupings (eg CU, DU, MEC, IMS, etc)
                • VNF
                  • Apps (like Gaming Servers)

In a real situation, you probably wouldn’t bother to model to this level of detail as it just makes more data to maintain. We’ve just included this detail to show some of the attributes of our sample 5G network.

5G also required some new templates, especially for core infrastructure that can house dozens of VNFs, 5G reference points and apps (eg games servers, CDN, etc) that you don’t want to recreate each time.

The 5G System architecture includes the following network functions (VNFs) and others.

  • Authentication Server Function (AUSF).
  • Access and Mobility Management Function (AMF).
  • Data Network (DN), e.g. operator services, Internet access or 3rd party services.
  • Unstructured Data Storage Function (UDSF).
  • Network Exposure Function (NEF).
  • Network Repository Function (NRF).
  • Network Slice Specific Authentication and Authorization Function (NSSAAF).
  • Network Slice Selection Function (NSSF).
  • Policy Control Function (PCF).
  • Session Management Function (SMF).
  • Unified Data Management (UDM).
  • Unified Data Repository (UDR).
  • User Plane Function (UPF).
  • UE radio Capability Management Function (UCMF).
  • Application Function (AF).
  • User Equipment (UE).
  • (Radio) Access Network ((R)AN).
  • 5G-Equipment Identity Register (5G-EIR).
  • Network Data Analytics Function (NWDAF).
  • CHarging Function (CHF).

Device Instances

We then create the devices to build the prototype network model shown in the first diagram above. This includes:

  • Hosting infrastructure
  • NFVI / VIM (NFV Infrastructure and Virtualised Infrastructure Management)
  • A 5GCN (5G Core Network)
  • An IMS (IP Multimedia Subsystem)
  • An RIC (RAN Intelligent Controller)
  • Virtualised Network Functions (AUSF, AMF, NRF, CU, DU, etc)
  • Mobile Edge Compute (MEC)
  • MEC Applications like gaming servers, CDN (Content Delivery Networks)
  • Radio Access Network (RAN) and Remote Radio Units (RRU)

This diagram below shows a small snapshot of the 5G Core. The templates we created earlier sure came in handy to avoid re-creating these hierarchies for each device type:

Note that the VirtualPorts are used for 5G reference points to support logical links, which we’ll cover later.

The diagrams below show the rack-layout views of core and edge hosting respectively. You’ll notice the hierarchy of device, NFVI, VNF-group, VNFs and applications are shown:

Physical Connections

To create the physical connectivity between core, edge and RRU, we’ve re-used the fibre cables, splice joints and ODFs that we demonstrated in the introduction to the OSS Sandpit inventory module.

In this case, we’ve just used fibres that were spare from last time and patched onto the 5G network’s physical infrastructure. The diagram below shows the physical path all the way from the Data Network (DN – aka a core router) to the transmitting antenna at site 2040.

This diagram includes router, core hosting, ODFs (optical patch panels), cables, splice joints, edge hosting, Radio Units and antenna, as well as fibre front and backhaul circuits.

Logical Connections

We also decided to create the various logical connections – in the most part these are interfaces between VNFs – via the standardised 5G Reference Points. 

You can also find a reference to the various logical interfaces / reference-points in the top-right corner of the prototype diagram (first diagram above).

You can also see the full list of reference points from any given VNF, as shown in the example of the AMF below. You’ll notice that these have already been set up as logical links to other components, as shown under “mplsLink” in the bottom pane. (ie the top pane are the “ports” on the AMF, the bottom pane shows the logical links to other VNFs)

The upper pane shows the instance of AMF (on the core) and its various interface points (A-end of the interface as VirtualPorts). The lower pane shows the relationships to Z-end components via logical circuits (note that I had to model them as MPLS links, which is not quite right, but the workaround needed in the tool).

You’ll also notice that the AMF is used by a number of network slices (under “uses” in the bottom pane), but we’ll get to that next.

Network Slices

Whilst not really technically correct, we’ve simulated some network slices in the form of “internal” services. To simplify, for each network slice type we’ve created a separate service terminating at each RRU. So, we’ve associated each RRU, Mobile Edge Infra (RAN), AMF (the Access and Mobility Management Function within the core) and the NSSF (the Network Slice Selection Function within the core) to these network slice “services.”

Some samples are shown below.

BTW 3GPP has defined the following Slice Types:

  • MIoT – Massive Internet of Things – support a huge device counts with enhanced coverage and low power usage
  • URLLC –  Ultra-Reliable Low-Latency Communications – to support low-latency, mission-critical applications
  • eMBB – Enhanced Mobile Broadband – to provide high speed data for application use (eg video conferencing, etc) and
  • V2X – Vehicle to Everything

Service Impact Analysis (SIA)

We can also use the service relationships to determine which Network Slices would be affected if the AMF failed. In the example below, there would be seven slices affected (see under “Uses” in the bottom pane), including all supported via sites 2040 and 2052

Similar analysis could be done using the getAffectedServices API that we demonstrated in the OSS Sandpit Inventory Intro post.

SigScale RIM

Over the last few weeks, I’ve also been using another open-source inventory management tool from SigScale called RIM (a Resource Inventory Manager designed to support service assurance use cases). It shines a light on mobile networks in particular.

The project creators authored the TM Forum best practice document IG1217 Resource Inventory of 3GPP NRM for Service Assurance which details the rationale for, and process of, mapping 3GPP information models to TM Forum’s TMF634 (Resource Catalog Mgmt) and TMF639 (Resource Inventory Mgmt) standards.

I plan to also use RIM’s REST interface (based on TM Forum’s OpenAPIs) to share data both ways with the Kuwaiba inventory module in the future. 

Summary

I hope you enjoyed this brief introduction into how we’ve modelled a sample 5G network into the Inventory module of our Personal OSS Sandpit Project. Click on the link to step back to the parent page and see what other modules and/or use-cases are available for review.

If you think there are better ways of modelling the 5G network, if I’ve missed some of the nuances or practicalities, I’d love to hear your feedback. Leave us a note in the contact form below.

OSS Sandpit – Resource / Inventory Module

This article provides a description of the inventory baseline, one module of our Personal OSS Sandpit Project.

As outlined in the diagram below, this incorporates the Inventory solution (by Kuwaiba), the graph database that underpins it as well as its APIs and data query tools. The greyed out sections are to be described in separate articles.

OSS Sandpit Inventory Baseline

We’ve tackled inventory first as this provides the base data set about resources in the network that other tools rely upon.

As the baseline introduction to the inventory module, we’ll provide a quick introduction to the following use-cases:

  • Building Reference Data like data hierarchies, device types, connectivity types, containment, device layouts, templates, flexible data models, etc
  • Creating Device Instances including rack views
  • Creating Physical Connections between devices
  • Creating Logical Connections between devices
  • Creating Services and their relationships with resources / inventory
  • Creating Outside Plant Views on geo-maps that include buildings, pits, splice cases, cable management, splicing, towers, antenna, end-to-end L1 circuits
  • Assigning IP Addresses and subnets with an IPAM tool
  • Creating an MPLS network
  • Creating an SDH network
  • Data import / export / updates via APIs including Service Impact Analysis (SIA)
  • Data import / export / updates via a Graph Database Query Language

Reference Data

Kuwaiba has a highly flexible and extensible data model. We’ve added many custom data classes (eg device categories like routers, switches, etc) such as those shown below:

And selectively added custom attributes to each of the classes (such as the Router class below):

Once the classes are created, we then create the Containment model (ie hierarchy of data objects). In our prototype, we’ve developed a custom containment model as follows:

  • Country
    • Site
      • System (Network Domain)
        • Rack
          • Equipment and so on.

We’ve also created a series of data templates to simplify data entry, such as the Cisco ASR 9001 and Generic Router examples below:

But we can also create templates for other objects, such as cables. The following sample shows a 24 fibre cable with two loose-tubes, each containing 12 fibre strands. (Note that colour-coding on tubes and strands is important for splicing technicians and designers)

Site and Device Instances

Next, we created some sites and devices within the sites, as shown below:

You’ll notice that some devices are placed inside a rack whilst others aren’t. You’ll also notice the naming convention for all devices (eg site – system – type – index, where site = 2052, system = DIS (distribution), type = CD (CD player for messaging) and index = 01 (the first instance of CD player at this site)).

It even allows us to show rack layouts (of equipment positions inside a rack)

And even patching-level details inside the rack (pink and blue lines represent patch-leads connecting to ports on the Cisco ASR 9001 router in rack position 2):

Physical Connections

Physical connections can take the form of patch-leads or via strands / conductors inside cables.

The diagram below represents a stylised optical fibre connection that we’ve created between a CODEC at site 2000 and another CODEC at site 2052. As you’ll also notice, it traverses two patch panels (ODFs – optical distribution frames), two splice joints and three optical fibre cables.

In our inventory tool, the stylised connection above presents as follows, where A and B have been added to indicate the patch-leads from the CODECs to patch-panels (ODFs):

Logical Connections

We can also represent logical and virtual connections. In the case below, we show a logical connection from the Waveguide of an antenna, to the broadcast of that signal to a neighbouring site, which then picks up the signal at the UAST (receiver).

Outside Plant Views

Outside plant (OSP) are the cables, joints, manholes, etc that help connect sites and equipment together. In the example below, we see the OSP view of the fibre circuits we described above in “Physical Connections.” If you look closely on the GIS (map overlay) below, you’ll spot sites 2000 and 2052, as well as the cables and splice joints. The lines show the physical route that the cables follow. 

You may also have noticed that the green line is showing a radio broadcast link, which is point-to-point radio and therefore follows a straight line path from antenna to antenna.

Cable Management

Cable management and splicing / connections is supported, with tubes/strands being selected and then terminated at each end of the cable (in this case CABLE1 and its strands connect the splice case on the left pane, with the ODF on the right pane). These can be managed on a strand-by-strand basis via the central pane. From the diagram, we can see that fibre 001 in CABLE 1 is connected to F1-001 in the splice case and 001-back on the ODF from the A-end and B-end details in the bottom left corner.

From the naming convention, you’ll notice that there are two sets of cable “ports” in the splice case, as indicated by fibre numbers starting with F1 and F2 respectively.

Topology Views

The diagram below shows a topological view of the devices within a site, helping operators to visualise connectivity relationships.

Services

One of the most important roles that inventory solutions play is as a repository of equipment and capacity. They also assist in allocating available resources to customer services. In the example below, service number “2052-ABC_LR_97.3FM-BSO” has a dependency on a tower, antenna, antenna switch frame and many more devices. If any of these devices fails, it will impact this customer service, as we’ll describe in more detail below.

 

IP Address Management (IPAM) and IP Assignment

We can manage IP address ranges / subnets, such as the examples below:

And then allocate individual IP addresses to devices, such as assigning IP address 222.22.22.1 to the CODEC, as shown on the “Physical Connection” diagram above.

MPLS Network

The following provides a simple MPLS network cloud for a customer:

APIs (including Service Impact Analysis Query)

The solution has hundreds of in-built APIs that facilitate queries, adds, modifies and deletes of data. 

The example shown below is getAffectedServices, which performs a service impact analysis. In this case, if we know that the device TEST-CD-02 fails, it will affect service number “2052-ABC_LR_97.3FM-BSO.” We can also look up the attributes of that service, which could include customer and customer contact details so that we can inform them their service is degraded and that repair processes have been initiated.

Note that the left-side pane is the Request and the right-side pane is the Response across the getAffectedServices API.

Data management via queries of the Graph Database

This inventory tool uses a Neo4j graph database. Using Neo4j browser, we can connect to the database and issue cypher queries (which are analogous to SQL, a structured query language that allows you to read/write data from/to the database). 

The screenshot below shows the constellation of linked data returned after issuing the cypher query (MATCH (n:InventoryObjects…. etc)). The data can also be exported in other formats, not just the graphical form shown here. 

I hope you enjoyed the brief introduction to the Inventory module of our Personal OSS Sandpit Project. Click on the link to step back to the parent page and see what other modules and/or use-cases are available for review.

Building a Personal OSS Sandpit

Being Passionate About OSS, I’ve used this blog / site to share this passion with the OSS community. The aim is to evangelise and make operational support tools even more impactful than they already are.

But there’s a big stumbling block – the barrier to entry into the OSS industry is huge. The barrier manifests in the following ways:

  • Opportunities – due to the breadth of knowledge required to be proficient, there aren’t many entry-level, OSS-related roles
  • Knowledge / Information – such is the diversity of knowledge, no single person has expertise across all facets of OSS – facets such as software, large-scale networks, IT infrastructure, business processes, project implementation, etc, etc. Even within OSS, there’s too huge a range of functional capability for anyone to know it all. Similarly, there’s no single repository of information, although organisations like TM Forum do a great job at sharing knowledge. On top of all that, the tech-centric worlds that OSS operate within are constantly evolving and proliferating
  • Access to ToolsOSS / BSS tools tend to be highly flexible, covering all aspects of a telco’s business operations (from sales to design to operations to build). OSS tend to cost a lot and take a long time to build / configure. That means that unless you already work for a telco or an OSS product vendor, you may struggle to get hands-on experience using the tools

It’s long been an ambition to help reduce the third barrier to entry by making personal OSS sandpit environments accessible to anyone with the time and interest.

The plan is to build a step-by-step guide that allows anyone to build their own small-scale OSS sandpit and try out realistic use-cases. The aim is to keep costs to almost nothing to ensure nobody is limited from tackling the project/s.

The building blocks of the sandpit are to be open-source and ideally reflect cutting-edge technologies / architectures. The main building blocks are:

  1. Network – a simulated, multi-domain network that can be configured and tested
  2. Fulfilment / BSS – the ability to create product offerings, take customer orders for those products, then implement as services into a network
  3. Assurance / Real-time – to perform (near) real-time monitoring of the network and services using alarms / performance / telemetry / logging
  4.  Resource / Inventory – to design and store records of multi-domain networks that spans PNI (Physical Network Inventory), LNI (Logical Network Inventory), OSP (Outside Plant), ISP (Inside Plant) and more
  5. Data Visualisation & Management – being able to interact with data generated via the abovementioned building blocks. Interact via search / queries, reports, dashboards, analytics, APIs and other forms of data import / export

OSS Sandpit Concept Diagram

Until recently, this sandpit has only been an ambition. But I’m pleased to say that some of these building blocks are starting to take shape. I’ll share more details in coming days and update this page.

This includes:

  • Introduction to The Resource / Inventory Module with the following use-cases:
    • Building Reference Data like location hierarchies, device types, connectivity types, containment, device layouts, templates, flexible data models, etc
    • Creating Device Instances including rack views
    • Creating Physical Connections between devices
    • Creating Logical Connections between devices
    • Creating Services and their relationships with resources / inventory
    • Creating Outside Plant Views on geo-maps that include buildings, pits, splice cases, cable management, splicing, towers, antenna, end-to-end L1 circuits
    • Assigning IP Addresses and subnets with an IPAM tool
    • Creating an MPLS network
    • Creating an SDH network
    • Data import / export / updates via APIs including Service Impact Analysis (SIA)
    • Data import / export / updates via a Graph Database Query Language
  • Designing the Inventory / Resources of a 5G Network including:
    • Hosting infrastructure
    • NFVI / VIM
    • A 5GCN (5G Core Network)
    • An IMS (IP Multimedia Subsystem)
    • An RIC (RAN Intelligent Controller)
    • Virtualised Network Functions (AUSF, AMF, NRF, CU, DU, etc)
    • Mobile Edge Compute (MEC)
    • MEC Applications like gaming servers, CDN (Content Delivery Networks)
    • Radio Access Network (RAN) and Remote Radio Units (RRU)
    • Outside Plant for fibre fronthaul and backhaul
    • Patching between physical infrastructure
    • End to end circuits between DN (Data Network), IMS, 5GCN, gNodeB, RRU
    • Logical Modelling of 5G Reference Points

More details and use-cases to come, including:

  1. Inventory modelling of:
    1. Satellite services
    2. Fixed wireless and Rural ISP network models
    3. Internet of Things (IoT)
    4. GPON / FTTH
    5. HFC / CableCo
    6. Data Centre (DC)
    7. MPLS
    8. SDH Transmission
    9. More network virtualisation (SDN), in addition to the virtualisation scenarios covered in the 5G prototype (see above)
    10. Are there any other scenarios you’d like to see???

 

If you’d like to know more about our Personal OSS Sandpit Project, fill in the contact form below.

Industry News: Vodafone NZ signs with Comarch

Breaking news: “Vodafone NZ signs with Comarch” has been published on our Industry News stream.

Industry News includes: contract wins, new product releases, job openings, EOIs/RFPs, etc.

To publish news about your organisation, first claim or register your organisation’s listing on The Blue Book OSS/BSS Supplier Directory then create a news post.

Industry News: Microsoft and Telstra to partner on cloud, IoT, and digital twins

Breaking news: “Microsoft and Telstra to partner on cloud, IoT, and digital twins” has been published on our Industry News stream.

Industry News includes: contract wins, new product releases, job openings, EOIs/RFPs, etc.

To publish news about your organisation, first claim or register your organisation’s listing on The Blue Book OSS/BSS Supplier Directory then create a news post.

Industry News: First to use SpaceX’s Starlink internet in the field: ‘It’s amazing’

Breaking news: “First to use SpaceX’s Starlink internet in the field: ‘It’s amazing’” has been published on our Industry News stream.

Industry News includes: contract wins, new product releases, job openings, EOIs/RFPs, etc.

To publish news about your organisation, first claim or register your organisation’s listing on The Blue Book OSS/BSS Supplier Directory then create a news post.

OSS/BSS in the Clouds, updated again

We posted an article in July entitled “OSS / BSS in the clouds,” which looked at the OSS, BSS and related telco infrastructure platforms being offered by AWS, Google, Microsoft and their partners. This followed a number of recent announcements made by the hyperscalers relating to their bigger pushes into telco. It had a particular focus on 5G models and the edge compute technologies that support them. 

After adding VMware to the list, we were also asked to include Red Hat (by interested onlookers, not IBM 😉 ), which we’ve now included. We’ve also incorporated recent announcements by Microsoft, including a view of their telco offerings.

OSS/BSS in the clouds, updated

We posted an article in July entitled “OSS / BSS in the clouds,” which looked at the OSS, BSS and related telco infrastructure platforms being offered by AWS, Google, Microsoft and their partners. This followed a number of recent announcements made by the hyperscalers relating to their bigger pushes into telco. It had a particular focus on 5G models and the edge compute technologies that support them. 

Following VMware’s announcement of its 5G Telco Cloud Portfolio on 1st Sept 2020, we’ve now retro-fitted VMware into the earlier post. Click on it here “OSS / BSS in the clouds,” to view the updated article.

Is omni-channel more disadvantage than advantage for telcos?

In our post on Monday, we discussed how some commodity providers have a structural advantage through lower cost of production (eg Rio Tinto in iron ore). Telcos have the potential to achieve a similar advantage on their commodity services too.

It also mentioned that the first principle behind that advantage is simplicity (of systems, overheads, processes, offerings, etc). Many of these simplification factors are controllable by our OSS/BSS, but most just end up with us as a result of up-stream decisions that flow to us for resolution. These decisions (eg running 50+ mobile offering variants including grandfathered services) make our OSS/BSS far more complex.

Another example is omni-channel customer engagement, which includes:

  • Call centres
  • Digital / websites
  • Email
  • SMS
  • Chatbots
  • Mobile Apps
  • IVR (Interactive Voice Response)
  • Social channels
  • Smart-home assistants
  • USSD (Unstructured Supplementary Service Data)

I completely get that we want to allow customers (or potential customers) to choose the contact mechanism/s they’re most comfortable with. Tight coupling / orchestration across these channels is important for customer loyalty.

Unfortunately, the task of coordinating all of these systems is complex. Linking and tracking customer journeys through multiple channels is even more challenging. For example, websites and IVR channels don’t require customers to self-identify, making it difficult to cross-link with in app touch-point data where identification is inherent. Some flows are personalised, some are interactive, some are batched. 

Gaining consolidated insights from these disparate data sources is imperative for customer satisfaction. 

Clearly a typical telco omni-channel story doesn’t represent simplicity, especially for the OSS/BSS that support it! That presents a barrier to achieving the cost of production advantage we discussed.

How do telcos compare with Apple, Amazon, Google and others? Do they offer as many customer contact channels? Do they have as many variants in their “extended supply chain?” Do they have as many potential fall-out points in their O2A, T2R, P2O, etc workflows? It appears to me that most telcos are at a structural disadvantage here (on costs and CX).

We’d love to hear your thoughts. Please leave us your perspectives in the comments section below.  

Is scaled OSS/BSS multi-tenancy a thing?

We talked yesterday about the commoditisation of telco services and the part that OSS/BSS have to play in differentiation. We also talked about telcos retaining a few competitive advantages despite the share-of-wallet inroads made by OTT, software and cloud service providers recently. Managed services is one area where some of those advantages converge.

Quite a few years ago I worked with one of Australia’s largest telcos on a managed service account for one of Australia’s big four banks. The value of the managed service contract was worth a couple of billion dollars. It covered the whole gamut of what a telco can offer. Voice, data, mobility and unified communications of course, but also branch / office comms fit-outs, international WANs, IT services, security, custom help-desk, dedicated service delivery management and much more.

Big companies tend to prefer to deal with big companies, especially when the services are complex and wide-ranging. A bank like this one could realistically only negotiate with a handful of telcos because only a few could provide viable size and scope to meet their needs… if they wanted to keep it to a consolidated procurement event.

This bank wasn’t this telco’s only large managed services contract. They had quite a few similar clients. The deals were all incredibly cut-throat between the other big telcos, but they were still profitable, especially once the variations started coming in. The telcos are not in a race to the bottom on managed services like the commodity data services we discussed yesterday.

I found it interesting that the telcos mainly focused on providing these managed service deals to the ASX200 (Australia’s top 200 companies by market capitalisation), give or take. So they service the top 200 (ish) with managed services and the long, long tail with retail services. But of course mid-market companies fall somewhere in the middle – big enough to require custom solutions, but too small to warrant the type of managed services the bank was getting.

So, now let me put the OSS/BSS spin on this situation (yes, it took me a while). At this telco, each of its big managed service contracts had completely bespoke OSS/BSS/portal solutions. Of course they had a core OSS/BSS – the factory that handled assurance and fulfilment for customers large and small – but every managed service had completely unique, satellite OSS/BSS/portals at their customer interface. For example, the core ran eTOM, the customer interface often demanded ITIL.

This telco couldn’t offer managed services to the mid-market because it was too expensive to set up the unique tooling. That meant there were almost no repeatability or economies of scale in processes, reporting, data science, skill portability, etc.

It bewildered me that they hadn’t invested the time into creating a cookie-cutter approach to all these satellite OSS/BSS/portals. Sure, each would need some customisation, but there was also significant potential for repeatability / consistency.

Now this may sound a bit bewildering to you too. But it’s not just the telco’s short-term thinking that leads to this situation. Most OSS/BSS vendors don’t build products with this type of multi-tenancy in mind.

The ability to spin up/down satellite OSS/BSS/portals, each with configurability but consistency, is not common. Importantly, nor is secure partitioning that ensures each customer only sees their data even when using shared infrastructure (including NOC / WOC / SOC as well as network infra and core OSS/BSS).

You know, I don’t recall ever hearing anyone else talk about this type of use-case. Multi-tenancy, yes, to an extent, but not at managed services scale.

Is this even a thing? Should it be? Could it be? You be the judge! I’d love to hear your thoughts / experiences. Please leave a comment below if your OSS/BSS stack supports this use-case or if your telco has developed a cookie-cutter approach. What approach do you use?

OSS’s Influence on Cost of Production

Since widespread deregulation of telecommunications globally, the passing of data has become a commodity. Perhaps it always was, but increased competition has steadily driven down dollar per bit. It’s likely to continue on that path too. Meanwhile the expected throughputs and consumption of data services is ramping ever-upwards, which requires investment in networks by their operators.

By definition, a commodity is a product/service that is indistinguishable between providers. The primary differentiator is price because there are no other features that make a buyer choose one provider over another. 

At face value, that’s true of commodities such as oil or iron ore, just as it is for data. What could be less differentiated than ones and zeroes?

However, as the charts below show for oil and iron ore, commodities aren’t always a level playing field. Some suppliers have significant differentiation – via cost of production advantages over their competitors.

Cost of Oil Production

Iron ore cost of Production

What if we created a $/bit graph of telcos similar to the oil graph above showing opex and capex splits? Which countries / operators would have significant advantage? Which telcos are like Rio Tinto, comfortable in the knowledge that no matter how low the spot price goes, their competitors will be deeply unprofitable (and possibly out of business) long before they will be.

Virtualisation of network infrastructure (SDN, NFV, cloud) has been the much-hyped saviour of telco business models. However, like me, you’ve probably started hearing news that savings from these architectures just aren’t eventuating. Cost-bases are possibly even going up because of their increased complexity. Either way, incremental cost reduction tends to have diminishing returns anyway.

It would seem that telcos are left with two / three choices:

  1. Have structurally lower production costs, like Kuwait and Rio Tinto; or
  2. Differentiate / Innovate, limiting exposure to the raw data / connectivity services that so many telcos still rely upon
  3. Or both

OSS/BSS has a part to play with both of these options.

Structurally lower cost comes not from cost reduction but from having far simpler variant trees (ie smaller numbers of product offerings, network types, system integrations, service configuration styles, support models, obsolescence of legacy / grandfathered products, minimal process options, etc, etc). Some of this happens within the OSS/BSS, but a lot more of it stems from upstream decisions.

Differentiation / innovation means being able to innovate through experiences / insights / content / trust / partnerships / ecosystems / local-presence in ways that other organisations can’t. It’s unlikely to be in software alone or in cloud infrastructure because others have proven to do that far more effectively than telcos. As much as they wish otherwise, it’s just not in the DNA of many. Yet that’s where most attention seems to be. Meanwhile OSS/BSS are waiting to be the glue that can leverage the competitive advantages that telcos do still hold.

How fragmentation is harming the OSS/BSS industry

Our Blue Book OSS/BSS Vendors Directory provides a list of over 400 vendors. That clearly states that it’s a highly fragmented market. This amount of fragmentation hurts the industry in many ways, including:

  • Duplication – Let’s say 100 of the 400 vendors offer alarm / fault management capabilities. That means there are 100 teams duplicating effort in creating similar functionality. Isn’t that re-inventing the wheel, again and again? Wouldn’t the effort be better spread into developing new / improved functionality, rather than repeating what’s already available (more or less). And it’s not just coding, but testing, training, etc. The talent pool overlaps on duplicated work at the expense of taking us faster into an innovative future
  • Profit-share – The collective revenues of all those vendors need to be spread across many investors. Consolidated profits would most likely lead to more coordination of innovation (and less duplication above). And just think at how much capability has been lost in tools developed by companies that are no longer commercially viable
  • Overhead – Closely related is that every one of these organisations has an overhead that supports the real work (ie product development, project implementation, etc). Consolidation would bring greater economies of scale
  • Consistency – With 400+ vendors, there are 400+ visions / designs / architectures / approaches. This means the cost and complexity of integration hurts us and our customers. The number of variants makes it impossible for everything to easily bolt together – not withstanding the wonderful alignment mechanisms that TM Forum, MEF, etc create (via their products Frameworx, OpenAPIs, MEF 3.0 Framework, LSO, etc). At the end of the day, they create recommendations that vendors can interpret as they see fit. It seems the integration points are proliferating rather than consolidating
  • Repeatability and Quality – Repeatability tends to provide a platform for continual improvement. If you do something repeatedly, you have more opportunities to refine. Unfortunately, the bespoke nature of OSS/BSS implementations (and products) means there’s not a lot of repeatability. Linus’s Law of OSS defects also applies, with eyeballs spread across many code-bases. And the spread of our variant trees means that we can never have sufficient test coverage, meaning higher end-to-end failure / fall-out rates than should be acceptable 
  • Portability – Because each product and implementation is so different, it can be difficult to readily transfer skills between organisations. An immensely knowledgeable, talented and valuable OSS expert at one organisation will likely still need to do an apprenticeship period at a new organisation before becoming nearly as valuable
  • Analysis Paralysis – If you’re looking for a new vendor / product, you generally need to consider dozens of alternatives. And it’s not like the decisions are easy. Each vendor provides a different set of functionality, pros and cons. It’s never a simple “apples-for-apples” comparison (although we at PAOSS have refined ways to make comparisons simpler). It’s certainly not like a cola-lover having to choose between Coke and Pepsi. The cost and ramifications of an OSS/BSS procurement decision are decidedly more significant too obviously
  • Requirement Spread – Because there are so many vendors with so many niches and such a willingness to customise, our customers tend to have few constraints when compiling a list of requirements for their OSS/BSS. As described in the Lego analogy, reducing the number of building blocks, perhaps counter-intuitively, can actually enhance creativity and productivity
  • Shared Insight – Our OSS/BSS collect eye-watering amounts of data. However, every data set is unique – collection / ETL approach, network under management, product offerings, even naming conventions, etc. This makes it challenging to truly benchmark between organisations, or share insights, or share seeded data for cognitive tools

However, I’m very cognisant that OSS come in all shapes and sizes. They all have nuanced requirements and need unique consideration. Yet many of our customers stand on a burning platform and desperately need us to create better outcomes for them.

From the points listed above, the industry is calling out for consolidation – especially in the foundational functionality that is so heavily duplicated – inventory / resource management, alarms, performance, workflows, service ordering, provisioning, security, infrastructure scalability, APIs, etc, etc. 

If we had a consistent foundation for all to work on, we could then more easily take the industry forward. It becomes a platform for continuous improvement of core functionality, whilst allowing more widespread customisation / innovation at its edges.

But who could provide such a platform and lead its over-arching vision?

  • I don’t think it can be a traditional vendor. Despite there being 400+ vendors, I’m not aware of any that cover the entire scope of TM Forum’s TAM map. Nor do any hold enough market share currently to try to commandeer the foundational platform
  • TM Forum, wouldn’t want to compromise their subscriber base by creating something that overlaps with existing offerings
  • Solution Integrators often perform a similar role today, combining a multitude of different OSS/BSS offerings on behalf of their customers. But none have a core of foundational products that they’ve rolled out to enough customers to achieve critical mass
  • I like the concept of what ONAP is trying to do to rally global carriers to a common cause. However, its size and complexity also worries me. That it’s a core component of the Linux Foundation (LF Networking) gives it more chance of creating a core foundation via collaborative means rather than dictatorial ones

We’d love to hear your thoughts. Is fragmentation a good thing or a bad thing? Do you suggest a better way? Leave us a message below.

A new revenue line just waiting for OSS/BSS to grab

I’m assuming that if you’re reading this blog, chances are you’re already an OSS/BSS expert, or spend a lot of your working life thinking about them. Perhaps you do more than think about them and actually help to implement them in some way. Perhaps you don’t implement them yet, but have been tasked with understanding them in more detail.

During those activities do you spend much time thinking about the end user (EU)?

But I guess I should first start by classifying who I think our EUs actually are. They can fall into a few different categories:

  • Internal EUs (IEUs) – If you’re developing an OSS/BSS in-house, then you might be providing a product / service to your colleagues in network operations, IT, sales, etc that helps them do their job
  • Client EUs (CEUs) – Similar to IEUs, but you’re providing a product / service to a client’s team in network operations, IT, sales, etc. This generally implies that you’re an OSS/BSS vendor or integrator that is supplying a solution to a client like a telco, utility or enterprise customer. Your solution helps them to provide a network-related service
  • Ultimate End Users (UEUs) – These are the people who consume network services. The IEUs and CEUs simply provide support to the UEUs to ensure their network and related services are all operational and usable.

My gut feel is that we don’t tend to think about UEUs very often. After all, I sometimes wonder whether our clients (eg telcos) only think about them in terms of how to avoid them.

The UEU calls the help line because they want someone to help them. But that’s expensive for our clients (eg telcos), so they’d rather:

  • IVR them or
  • Chat-bot them or
  • Point them to canned URLs / FAQs or
  • App them

When you are the upset customer, you want a full, uninterrupted hearing, you want to deal with someone with the authority to fix the problem and you want a fair resolution. You don’t want to be sent a copy of the company’s warranty policy.”
Jeffrey J. Fox.

The “typical” approach to handling upset customers (UEUs) for our telco clients is akin to sending a copy of the company’s warranty policy.

But the telcos are in a bit of a catch-22 situation. They want to keep customers happy (ie they don’t want churn). But they generally don’t have the tools that give sufficient insights about why the customer is upset. An expensive call centre operator (CEU) often adds little extra value than the cheap customer avoidance channels dot-pointed above. So customer avoidance mechanisms are invested in.

For example, NOC operators (CEUs) can see logs, alarms, performance within their network, but that doesn’t often directly translate into a user’s experience. It certainly doesn’t translate the network telemetry into words / actions that a contact centre operative (CEU) can clearly communicate with UEUs (especially non-tech-savvy UEUs).

Personally, I think this is a massive opportunity for OSS/BSS developers. If we could create the tools that:

  1. Could reliably interpret real UEU experiences (ie health of the service as experienced by the UEU, not reverse-engineering nodal info and guessing what the experience might be)
  2. Diagnose degradations / failures / fallouts / problems in UEU services
  3. Translate the diagnosis into actions / recommendations – either for the UEU to perform as self-help, or for CEU / backend-systems to repair
  4. Inform UEU of what’s happening and when the problem will be solved
  5. If the customer calls before a proactive notification can be sent, ensure the collective telemetry insights / recommendations are available for contact centre operators (CEU) to help resolve rather than deflect the problem

… then there would be less incentive for building the annoying customer avoidance mechanisms. If we can do that, OSS/BSS would surely pick up more of the investment that’s currently carved out for chat-bots, IVR, etc. It’s a new/improved revenue line, but only because it better helps the people further down the line who ultimately pay our bills.

Bollinger bands and candlestick charts in OSS

No doubt all of you have seen network performance graphs. The one below is an example (from Flowmon 8.03). This example shows throughput, jitter and round-trip time amongst other metrics. No doubt you use many additional metrics to track the health of your network.

Network Performance Graph

Most performance management tools show the range of metrics as line or bar graphs, as above, which is helpful of course. They often also have threshold-crossing alert capabilities. For example, if utilisation crosses a certain threshold (eg 70%), then send an alert to do something about it. Or even better, trigger an automation to do something about it.

The only problem of using this approach is that the discrete values you see are not telling the whole story. You’re generally seeing the averaged value for a given poll-cycle (eg 5 mins), not the fluctuations that appeared during the poll-cycle.

That got me thinking that candlestick charts might also show some useful information when visualising network performance. Candlesticks, such as the one shown below, are more often used for financial analysis.

They show the open and close (for a given recording period) in the thick body of the “candle” (red for a drop, green for a rise); as well as the high and low during the recording period in the candle wicks.

This gives you a much better feel for the fluctuations during a given recording period. However, in my many years in OSS/BSS, I’ve yet to see this technique used. Have you?

Whilst discussing the possible merits of this technique with a good friend and super-talented OSS/BSS developer, I was alerted to an additional concept – Bollinger Bands. Hat-tip Jay!

Bollinger Bands are also generally used in financial analysis. They are the lines marked in blue, overlaid onto the candlestick diagram below. They’re similar to moving average envelopes, but include standard deviations in their calculation. They’re useful because they identify periods of volatility such as the yellow box in the diagram below. The yellow box has identified abnormal behaviour well before a typical threshold-crossing event would have occurred, thus giving advanced warning. 

Generally speaking, performance is tending towards upper limits when the candlestick is near the upper blue line and tending towards lower limits when near the lower blue line.

What do you think? Could candlesticks and Bollinger Bands help you keep your network within acceptable performance bounds?

BTW. Time can be an important factor for monitoring network health trending, especially since 5, 10 and 15 minute poll cycles are still common. Let’s say the start of the 15m poll cycle is at 00:00, so the EMS/NE performance for 15 mins to then logs the record at 00:15 (or perhaps later depending on its ETL efficiency – let’s say 00:20). The log then gets pulled and processed and visualised by a performance visualisation tool, which requires further time (let’s say 10 mins). So now it’s 00:30 before any person or system can react to what started happening 30 mins earlier. Most performance graphs just show the average value over each 15m poll period. If your EMS/NE can also export max and min during each period, you’ll have the ability to generate a candlestick. Then in turn you can generate your Bollinger Bands. It doesn’t speed up the delays mentioned above, but it will give you additional granularity on a single graph though.

Note: Candlesticks are similar to box plots, which are also used to show variability within time-series data. A box plot (aka box and whisker chart) shows:

  1. Outliers (optional)
  2. Upper end of range excluding outliers
  3. Lower end of range excluding outliers
  4. First and Third quartile marks
  5. Median

Hat-tip to Jim for recommending box plots.

OSS / BSS in the clouds

Have you noticed the recent up-tick in headlines around telco offerings by hyperscalers AWS, Google and Microsoft? Or the multi-cloud telco models, the middleware, supplied by VMware and Red Hat?

Whilst previous generations of wireless connectivity have focussed on voice and data capabilities, 5G is architected to better enable consumer business models. Edge compute (both on-prem / device-edge and provider edge) and related services to support 5G use-cases appears to be the leading driver behind recent announcements.  These use-cases will need to be managed by our OSS/BSS for the telco operators and their customers.

Meanwhile, top-tier OSS/BSS users are also continuing to adopt cloud-native OSS/BSS initiatives, as described in this Infographic from Analysis Mason / Amdocs. Analysis Mason estimates that over 90% of CSPs in North America, Asia–Pacific and Europe will have their OSS/BSS stacks running on cloud infrastructure by 2022, with well over 60% on hybrid cloud.

However, just how much of the CSP OSS/BSS stack will be on the cloud remains in question. According to TM Forum’s research, most CSPs have deployed less than 5% percent of their operations software in the public cloud.

In today’s article, we take a closer look into cloud offerings for OSS/BSS. The providers we’ll cover are hyperscalers:

AWS

https://aws.amazon.com/telecom/

The following diagrams come from the Amazon Telco Symposium. The first diagram shows the AWS Telecom Engagement Model (noting the OSS/BSS bubble).

The latter diagram provides some insight into important offerings in AWS’ push into the 5G / telco edge such as Greengrass, SiteWise, Sagemaker and more.

 

AWS services such as the following have been used as part of home-grown offerings for years:

  • Wavelength (low latency), Lambda (serverless) or EC2 – compute services for processing applications/code
  • S3, EFS, Glacier, Elastic, Snow Family, etc – data storage for collecting logs, etc
  • EKS or ECS – for Kubernetes / Docker container / cluster management
  • VPC – for separate environment deployments
  • VPN – to tie VPCs to networks / clouds / DCs
  • ELB – for load balancing
  • ELK – for log management consisting of three open source projects: Elasticsearch, Logstash, and Kibana
  • Aurora, RDS, Redshift, DynamoDB, Neptune, KDB, etc – databases
  • Cassandra, Kibana, etc – data visualisation
  • SageMaker, Augmented AI, Lex, etc – AI / ML tools
  • And much more

These have been leveraged by telco architects to build home-grown OSS/BSS tools that leverage commercial and open-source products like Apache’s Kafka, NiFi, Spark, etc.

However, there’s been an increasing trend for OSS/BSS vendors to publish their offerings on the AWS marketplace too, including:

  • Moogsoft
  • Zoho / ManageEngine (eg OpManager, Network Config Mgr)
  • Solarwinds
  • Domotz Pro
  • Lumeta CloudVisibility
  • Flowmon
  • Hyperglance
  • Mphasis InfraGraf (Forecasting & Planning)
  • KloudGin (Work Order Mgmt)
  • Kx (network performance)
  • AND Bosch, ThingsBoard, ThingPark, ThingLogix, etc, etc (if we extend into IoT device management)

AWS Marketplace tends to show the solutions that are more standardised / fixed-price in nature (Telecoms section in Marketplace). Many other OSS/BSS vendors such as Netcracker, CSG, Intraway and Camvio don’t appear in the AWS marketplace but have customisable, AWS-ready solutions for clients. These companies have their own sales arms obviously, but also train the AWS global salesforce in their products.

Google Cloud

https://cloud.google.com/solutions/telecommunications

According to Google Cloud’s strategy for the telecom industry, Google Cloud is focusing on three strategic areas to support telecommunications companies:

  • Helping telecommunications companies monetise 5G as a business services platform, including:

    • The Global Mobile Edge Cloud (GMEC) strategy, which will deliver a portfolio and marketplace of 5G solutions built jointly with telecommunications companies; an open cloud platform for developing network-centric applications; and a global distributed edge for deploying these solutions

    • Anthos for Telecom, which will bring its Anthos cloud application platform to the network edge, allowing telecommunications companies to run their applications wherever it makes the most sense. Anthos for Telecom—based on open-source Kubernetes—will provide an open platform for network-centric applications.
  • Empowering telecommunications companies to better engage their customers through data-driven experiences by:

    • Empowering telecommunications companies to transform their customer experiences through data- and AI-driven technologies. Google’s BigQuery platform provides a scalable data analytics solution—with machine learning built-in so telecommunications companies can store, process, and analyze data in real time, and build personalization models on top of this data

    • Contact Center AI assists telecommunications companies with customer service. Contact Center AI gives companies 24/7 access to conversational self-service, with seamless hand-offs to human agents for more complex issues. It also empowers human agents with continuous support during their calls by identifying intent and providing real-time, step-by-step assistance
    • AI and retail solutions including omni-channel marketing, sales and service, personalisation and recommendations, and virtual-agent presence in stores
  • Assisting them in improving operational efficiencies across core telecom systems. This allows operators to move OSS, BSS and network functions from their own environments to the Google Cloud

This LightReading report even highlights how Google has been engaged to provide extensive knowledge transfer to some telcos.

This press release from March 2020 announced that Google would partner with Amdocs to support the telecom industry to:

  • Deliver Amdocs solutions to Google Cloud: Amdocs will run its digital portfolio on Google Cloud’s Anthos, enabling communications service providers (CSPs) to deploy across hybrid and multi-cloud configurations
  • Develop new enterprise-focused 5G edge computing solutions: Amdocs and Google Cloud will create new industry solutions for CSPs to monetize over 5G networks at the edge
  • Help CSPs leverage data and analytics to improve services: Amdocs will make its Data Hub and Data Intelligence analytics solutions available on Google Cloud. Amdocs and Google Cloud will also develop a new, comprehensive analytics solution to help CSPs leverage data to improve the reliability of their services and customer experiences.
  • Partner on Site Reliability Engineering (SRE) services: The companies will share tools, frameworks, and best practices for SRE and DevOps

On the same day the Google / Amdocs partnership was announced, Netcracker Technology announced it would deploy its entire Digital BSS/OSS and Orchestration stack on Google Cloud. These applications are cloud native, deployed as a set of reusable microservices that run over on-prem or public cloud on top of container platforms such as Google Kubernetes Engine (GKE).

Optiva (formerly Redknee) has also adopted a Google Cloud strategy, using Google Cloud Spanner and the Google Cloud Platform (GCP) to underpin its Charging Engine.

This article from CIMI Corp provides some great additional reading about why Google is a credible telco cloud provider.

Microsoft Azure

https://www.microsoft.com/en-us/industry/telecommunications

Microsoft has also announced an intention to better serve telecom operators at the convergence of cloud and comms networks through its Azure platform.

The diagram below, from this Microsoft blog, shows their coverage map of offerings for CSP customers (Azure for Operators):

Microsoft's Telco Offering Coverage Map

The blog also indicates that their offering is built upon:

  • Interconnect – 170 points of presence and 20,000 peering connections globally. More than 200 operators are already integrated with the Azure network via their ExpressRoute service
  • Edge – offered by the Azure platform, whether at enterprise edge, network edge, network core or cloud
  • Network Functions – This is where Microsoft distinguishes itself from AWS and Google. Its ability to offer network, particularly for 5G via RAN and Mobile Packet Core offerings (see more about Microsoft’s Affirmed and Metaswitch acquisitions below)
  • Cloud – Incorporates a marketplace with capabilities including OSS/BSS, IoT (via IoT Central), Machine Learning / AI, Azure Cognitive Services (APIs/SDKs that help developers build cognitive intelligence across Decisions, Vision, Speech, Language and Web-search)

“We will continue to partner with existing suppliers, emerging innovators and network equipment partners to share roadmaps and explore expanded opportunities to work together, including in the areas of radio access networks (RAN), next-generation core, virtualized services, orchestration and operations support system/business support system (OSS/BSS) modernization,” states Yousef Khalidi in this Microsoft post.

Acquisition of Metaswitch Networks (a provider of virtualised network software) and Affirmed Networks (a provider that sells virtualised, cloud-native mobile network solutions) shows further evidence of ambitions in the telco / cloud domain.

Like the partnerships described with AWS and Google above, Netcracker has also partnered with Microsoft, offering its Netcracker Digital BSS/OSS and Orchestration applications on Microsoft Azure. This article also describes that, “Netcracker is collaborating with Microsoft to integrate Azure Machine Learning (ML) and AI services with Netcracker’s Advanced Analytics to add intelligent contextual decisioning and recommendations to enable more personalized customer engagements.”

Meanwhile, Amdocs and Microsoft have been working on making ONAP available on the Azure platform. Nokia and Microsoft are partnering on “…cloud, Artificial Intelligence (AI) and Internet of Things (IoT), bringing together Microsoft cloud solutions and Nokia’s expertise in mission-critical networking.”

Microsoft Azure Edge Zones are offered through Azure, with select carriers and operators, or as private customer zones. They bring compute, storage, and service availability closer to the customer / device for low-latency + high-throughput use cases.

The recent announcement of an AT&T and Microsoft alliance as well as deals involving Telefónica (with Aura, its AI-powered digital assistant), SK Telecom (5G-based cloud gaming), Reliance Jio (cloud solutions), NTT (enterprise solution offerings), and Etisalat (future networks) show an increasing presence for Azure within the telco domain.

Netcracker has integrated its solution with Microsoft business applications (eg Office 365, Dynamics 365 and OneDrive), as other OSS/BSS providers undoubtedly have too.

Microsoft also has their own OSS/BSS offering in Dynamics 365 Field Service Management.

VMware Telco Cloud

telco.vmware.com 

VMware telco cloud architecture

VMware’s recently announced 5G Telco Cloud Portfolio has been designed to give network operators the platform to accelerate 5G and Edge implementation. Its key differentiator from the examples provided above is it allows operators to run containerised workloads across private, telco, edge and public clouds. This is seen as being an important feature allowing telcos to avoid cloud partner lock-in.

The press release above indicates that, “VMware is evolving its VMware vCloud NFV solution to Telco Cloud Infrastructure, providing CSPs a consistent and unified platform delivering consistent operations for both Virtual Network Functions (VNFs) and Cloud Native Network Functions (CNNFs) across telco networks. Telco Cloud Infrastructure is designed to optimize the delivery of network services with telco centric enhancements, supporting distributed cloud deployments…
Tightly integrated with Telco Cloud Infrastructure, VMware’s Telco Cloud Automation intelligently automates the end-to-end lifecycle management of network functions and services to simplify operations and accelerate service delivery while optimizing resource utilization. Telco Cloud Automation also now supports infrastructure and Containers-as-a-Service (CaaS) management automation to streamline workload placement and deliver optimal infrastructure resource allocation. It also significantly simplifies the 5G and telco edge network expansions through zero-touch-provisioning (ZTP) whenever capacity is required.

Red Hat

www.redhat.com/telco

Red Hat’s origins, of course, are in open-source tools such as Red Hat Linux. They’ve now evolved to become a leading provider of open-source solutions for enterprise, delivering Linux, cloud, container and Kubernetes technology. Red Hat was acquired by IBM in 2019.

Red Hat’s telco offerings are built upon the premise that service providers will use more open-source, multi-vendor solutions to underpin their OSS, BSS and networks of the future. Red Hat aims to offer open infrastructure to facilitate service provider initiatives such as NFV, 5G, OpenRAN and Edge Compute. This includes coordination of telco and IT infrastructure, but also the applications and data that are intertwined with them.

Red Hat’s telco proposition is supported by:

  • OpenShift – a cloud compute platform as a service (PaaS), built upon Docker containers (containerised applications) that are managed by Kubernetes. This is particularly relevant to support telco cloud models that provide virtualised network functions. It also helps to deliver the edge compute infrastructure that’s becoming synonymous with 5G
  • OpenStack – a set of components, mostly deployed as infrastructure as a service (IaaS) that help manage compute, storage and networks. Some of the components are shown in the diagram below sourced from redhat.com
    OpenStack Components (not complete)
  • Ansible Automation Platform – to automate network configuration, fault remediation, security updates, and more
  • Marketplace – to assist service providers in finding, buying and deploying certified, container-based software
  • Telco Ecosystem Program – that brings together enterprise and community partners to deliver integrated telco solutions. Partners include Affirmed, Altiostar, Atos, Cisco, Ericsson, Amdocs, MYCOM OSI, Zabbix, Metaswitch, Nokia, Juniper and more
  • Consulting – offering service resources that include Innovation Labs, training and consulting
  • And other solutions such as Ceph Storage, Cloud Suite, Quay, JBoss suite, Integration, Insights, Fuse and more.

Summary

CSPs (Communications Service Providers) find themselves in a catch-22 position with cloud providers. Their own OSS/BSS and those of their suppliers have an increasing reliance on cloud provider services and infrastructure. Due to economies of scale, efficiency of delivery, scalability and a long-tail of service offerings (from the cloud providers and their marketplaces), CSPs aren’t able to compete. Complexity of public cloud (security, scalability, performance, interoperability, etc) also make it a quandary for CSPs. It’s already a challenge (commercially and technically) to run the networks they do, but prohibitively difficult to expand coverage further to include public cloud. 

Yet, by investing heavily in cloud services, CSPs are funding further growth of said cloud providers, thus making CSPs less competitive, but more reliant, on cloud services. Telco architects are becoming ever more adept at leveraging the benefits of cloud. An example is being able to spin up apps without having to wait for massive infrastructure projects to be completed first, which has been a massive dependency (ie time delay) for many OSS/BSS projects.

In the distant past, CSPs had the killer apps, being voice and WAN data. These services supported the long-tail of business (eg salespeople from every industry in the world would make sales calls via telephony services) and customers were willing to pay a premium for these services.

The long-tail of business is now omni-channel, and the killer apps are content, experiences, data and the apps that support them. Being the killer apps, whoever supplies them also takes the premium and share-of-wallet. AWS, Google and Microsoft are supplying more of today’s killer apps (or the platforms that support them) than CSPs are.

The risk for CSPs is that cloud providers and over the top players will squeeze most of the profits from massive global investments in 5G. This is exacerbated if telco architects get their cloud architectures wrong and OPEX costs spiral out of control. Whether architectures are optimal or not, CSPs will fund much of the cloud infrastructure. But if CSPs don’t leverage cloud provider offerings, the infrastructure will cost even more, take longer to get to market and constrain them to local presence, leaving them at a competitive disadvantage with other CSPs

If I were a cloud provider, I’d be happy for CSPs to keep providing the local, physical, outside plant networks (however noting recent investments in local CSPs such as Amazon’s $2B stake in Bharti Airtel and Google’s $4.7 billion investment in Jio Platforms* not to mention Google Fiber and sub-sea fibre roll-outs such as this). It’s CAPEX intensive and needs a lot of human interaction to maintain / augment the widely distributed infrastructure. That means a lot is paid on non-effective time (ie the travel-time of techs travelling to site to fix problems, managing resources and/or coordinating repairs with property owners). Not only that, but there tends to be a lot of regulatory overhead managing local infrastructure / services as well as local knowledge / relationships. Governments want to ensure all their constituents have access to communications services at affordable prices. All the while, revenue per bit is continuing to drop, so merely shuffling bits around is a business model with declining profitability.

With declining profitability, operational efficiency improvements and cost reduction becomes even more important. OSS/BSS tools are vital for delivering improved productivity. But CSPs are faced with the challenge of transforming from legacy, monolithic OSS/BSS to more modern, nimble solutions. The more modular, flexible OSS/BSS of today and in future roadmaps are virtualised, microservice-based and designed for continuous delivery / DevOps. This is painting CSPs into a cloud-based future.

Like I said, a catch-22 for CSPs!

But another interesting strategy by Google is that its Anthos hybrid cloud platform will run multi-cloud workloads, including workloads on AWS and Microsoft Azure. Gartner predicts that >75% of midsize and large organisations will have adopted a multi-cloud and/or hybrid IT strategies by 2021 to prevent vendor lock-in. VMware (Dell) and Red Hat (IBM) are others creating multi-cloud / hybrid-cloud offerings. This gives the potential for CSPs to develop a near-global presence for virtualised telco functions. But will cloud providers get there before the telcos do?

For those of us supporting or delivering OSS/BSS, our future is in the clouds either way. It’s a rapidly evolving landscape, so watch this space.

 

* Note: Google is not the only significant investor in Jio:

Investor US$B Stake
Facebook 5.7 10%
Silver Lake Partners 1.43 2.08%
Mubadala 1.3 1.85%
Adia UAE Sovereign 0.8 1.16%
Saudi Arabia Sovereign 1.6 2.32%
TPG 0.64 0.93%
Catterton 0.27 0.39%
Intel 0.253 0.39%
Qualcomm 0.097 15.00%
Google 4.7 7.70%
TOTALS 16.79 42%

 

OSS/BSS Testing – The importance of test data

Today’s is the third part in a series about OSS/BSS testing (part 1, part 2).

Many people think about OSS/BSS testing in terms of application functionality and non-functional requirements. They also think about entry criteria / pre-requisites such as the environments, application builds / releases, test case development and maybe even the integrations required.

However, an often overlooked aspect of OSS/BSS functionality testing is the  data that is required to underpin the tests.

Factors to be considered will include:

  • Programmatically collectable data – this refers to data that can be collected from data sources. Great examples are near-real-time alarm and performance data that can be collected by connecting to the network, either to real devices and NMS or simulators
  • Manually created or migrated data – this refers to data that is seeded into the OSS/BSS database. This could be manually created or script-loaded / migrated. Common examples of this are inventory data, especially information about passive infrastructure like buildings, cables, patch-panels, racks, etc. In some cases, even data that can be collected via a programmatic interface still needs to be augmented with manually created or migrated data
  • Base data – for consistency of test results, there usually needs to be a consistent data baseline to underpin the tests.
  • Reversion to baseline – If/when there are test cases that modify the base data set (eg provisioning a new service that reserves resources such as ports on a device), there may need to be a method of roll-back (or reinstatement) to base state. In other cases, a series of tests may require a cascading series of dependent changes (eg reserving a port before activating a service). These examples may need to be rolled-back as a sequence
  • Automated / Regression testing – if automations and/or regression testing is to be performed, automated reversion to consistent base data will also be required. Otherwise discrepancies will appear between automated test cycles
  • Migration of base data – for consistency of results between phases, there also needs to be consistency of data across the different environments on which testing is performed. This may require migration of base data between environments (see yesterday’s post about transitions between environments)
  • Multi-homing of data – particularly for real-time data, sources such as devices / NMS may issue data to more than one destination (eg different environments).
  • Reconciliation / Equivalency testing – When multi-homing of data sources is possible or when Current and Future PROD are running in parallel, equivalency testing can be performed. By performing comparison between the processed data in destination systems it allows for equivalency testing (eg between current and future state mediation devices / probes / MDDs). Transition planning will also be important here as we plan to migrate from Current PROD to Future PROD environments
  • Data Migration Testing – this is testing to confirm the end-to-end validity and completeness of migrated data sets
  • PROD data cuts – once a system is cut over to production status, it’s common to take copies of PROD data and load it onto lower environments. However, care should be taken that the PROD data doesn’t overwrite specially crafted base data (see “base data” dot-point above)

 

OSS/BSS Testing – Transitions

One of the most vital, but underestimated aspect of OSS/BSS project implementation is ensuring momentum is maintained. These large and complex projects are prone to stagnating at different stages, which can introduce pressure onto the implementation team.

As mentioned in yesterday’s post, the first in this week’s series, the test strategy and scheduling is regularly overlooked as a means of maintaining OSS project momentum. More specifically, careful planning of transitions between test phases and the environments that they’re run on, can demonstrate progress – where progress is seen through the introduction of business value.

The following diagram provides a highly stylised indicative timeline (x-axis) of activities, showing how to leverage multiple different environments (y-axis). See here for examples of additional environments that you may have on your project.

You’ll notice that this diagram covers:

  • Environments
  • Test phases (eg FAT, SAT, SIT, DMT, NFT, UAT – see descriptions of these test phases here)
  • Build phases (build, configure, integrate)
  • Data loads (reference data, symbolic data and/or real data extracts)
  • How builds and data loads can be cascaded between environments to reduce duplicated effort

OSS Phasing - Testing, Environments, Data

Some other important call-outs from this stylised diagram include:

  • The Builds and Data Loads cascade from lower environments (eg from PROD-SUPPORT to PRE-PROD). Thought needs to be given as to which builds and data sets need to be cascaded from which environments
  • However, after PRE-PROD is handed over and becomes PROD, it is common for cuts of production data to be regularly loaded back into the lower environments so that they are PROD-like
  • Stand-up of new PROD environments is often a long lead-time item (because of size and complexity such as resilience architectures, security, etc), compared to lower environments. Environments such as DEV/TEST could be as easy to stand up as creating a new Virtual Machine/s on existing hosting
  • Different environments may have access to different data sources / integrations. For example, the lower environments may be connected to lab versions of devices and NMS/EMS. Alternatively, the network might be mimicked by simulators in non-PROD environments. Other integrations could be for active directory (AD), environment logging, patch management, etc. Sometimes production data sources can be connected to non-PROD OSS environments, but this is not so common
  • The item marked as Base-build on the PRE-PROD / PROD environment reflects the initial build and configuration of virtualisation, databases, storage, management networking, resilience / failover mechanisms, backup/restore, logging, security hardening and much more
  • Careful transition planning needs to go into PROD cutover. In the sample diagram, a final cut of data comes from the Current PROD environment to PRE-PROD before it becomes the Future PROD environment during official handover
  • You’ll notice though that there may still be a period of overlap between cutover from Current PROD to Future PROD. This is because there needs to be staged data source cutover in cases where data sources like network devices can’t multi-home data feeds to both environments in parallel.

This earlier post provides some insights into novel ways to slice and dice your OSS implementation by planning of regular drops and consistent release of business value

OSS/BSS Testing – the V-Model

On major software projects like the OSS you’re building, testing is an important phase of course. You’ll have undoubtedly incorporated testing into your planning. After all, testing is a key component of any Software Development Life Cycle (SDLC). There are various SDLC models / methodologies such as Waterfall, V-Model, Agile and others that you can consider.

Unfortunately, most OSS project teams tend to underestimate the testing phase, thinking it can just fit in around other major activities towards the end of the implementation. Experienced testers will suggest that they should be involved right from the requirement capture phase, because they’ll have to design test cases to prove that each requirement is met.

More importantly, your test strategy and test phase transitioning can play a major part in maintaining momentum through a project’s delivery phase. We’ll look into a number of related details in a series of posts this week.

Today we’ll look at the V-Model. It can be a helpful model for mapping requirements to test phases / cases. The diagram below, which comes from my book, Mastering your OSS, shows a simplified, sample version of the V-Model. It highlights the relationship between key test artefacts (eg plans / designs / specifications / requirements) on the left with the corresponding test phases on the right.

V-Model Testing

Your documentation and test phases will probably differ. You can find a discussion about some other possible OSS test phases here.

We’ll take a closer look tomorrow at how your different test phases could map to the OSS environments you might have available.

Getting confused by key Assurance metrics?

Are you a bit slow like me and sometimes have to stop and think to differentiate your key assurance metrics like your MTTRs from your MTBFs?

If so, I thought this useful diagram from researchgate.net might help

The metrics are:

MTBF (Mean Time Between Failures) – the average elapsed time between failures of a system, service or device. It’s the basic measure of availability / reliability of the system / service / device. The higher, the better.

MTTR (Mean Time to Repair) – generally used to denote the average time to close a trouble ticket (to repair a failed system / service / device). It’s the basic measure of corrective action efficiency. The lower, the better.

Some also use MTTR as a Mean Time to Recover / Resolve (ie MTTD + MTTR in the diagram above) or Mean Time to Respond (MTTD in the diagram above to acknowledge an event and create a ticket). See why I get confused?

MTTD (Mean Time to Detect / Diagnose) – the average time taken from when an event is first generated and timestamped to when the NOC detects / diagnoses the cause and generates a ticket. The lower, the better.

MTTF (Mean Time to Failure) – the average system / service / device up-time

The common data store trend (part 2)

Last month we posted an article that described using a common data model (CDM) for our OSS / BSS data. It mostly looked at the situation within the context of typical operational data sources (the blue boxes on the left side of the diagram below):

Today’s article pushes the vision a little further. If your CDM is built as part of an enterprise-wide data warehouse, then you may get the opportunity to think beyond the boundaries network and operations data.

We’ve long said that our OSS/BSS impact most other parts of a business, yet we tend to spend very little time proactively seeking value-add opportunities outside network operations, in marketing, products, finance, the C-suite and beyond. 

Traditional OSS/BSS were built around highly structured, relational databases, usually designed by OSS/BSS product vendors. Each data architecture was designed to support the specific, baked-in use-cases offered by each product. It was like a building architect designing a building, let’s say a new wing of a university, from the ground up for a very specific purpose.

You don’t get the same luxury with your CDM. You have to take a multitude of existing platforms, applications and data models and attempt to turn them into a cohesive data set. This is a bit like taking a row of existing houses, extending and combining them to form a university wing. The “existing houses” might represent disparate OSS or BSS or network systems, but they could also be IT / data silos from various other parts of the organisation.

You know the latter “university” design will be compromised – discrepancies in data standards, data flows, siloed data knowledge, disjointed data governance, etc. However, it also comes with a big benefit. You can keep appending new sets of data that were never part of initial considerations, of any of the IT / data silos. It could be weather data, social trending, building approvals or so much more. It could be any data set that you think could unlock new insights.

But to form a coherent and valuable data set, you still need a common blueprint. As Stephanie Shen describes on towardsdatascience.com

“…the following areas need to be considered and planned at this conceptual stage:

– The core data entities and data elements such as those about customers, products, sales.
– The output data needed by the clients and customers.
– The source data to be gathered and transformed or referenced to produce the output data.
– Ownership of each data entity and how it should be consumed and distributed based on business use cases.
– Security policies to be applied to each data entity.
– The relationships between the data entities, such as reference integrity, business rules, execution sequence.
– Standard data classification and taxonomy.
– Standards of data quality, operations, and Service Level Agreements (SLAs).
 
This conceptual level of design consists of the underlying data entities that support each business function.
 
As with all valuable OSS tripods, your value in the CDM chain is being able to connect the silos of business, IT and operations.