How and why we need to Challenge our OSS Beliefs

The OSS / BSS industry tends to be quite technical in nature. The networks we manage are technical. The IT stacks we build on are technical. The workflows we design are technical. The product offerings we coordinate are technical. As such, the innovations we tend to propose are quite technical too. In some cases, too technical, but I digress.

You would’ve no doubt noticed that the organisations garnering most attention in the last few years (eg the hyperscalers, Airbnb, Uber, Twitter, Afterpay, Apple, etc) have also leveraged technology. However, whilst they’ve leveraged technology, you could argue that it’s actually not “technical innovation” that has radically changed the markets they service. In most cases, it’s actually the business model and design innovations, which have simply been facilitated by technology. Even Apple is arguably more of a design innovator than a technology innovator (eg there were MP3 players before the iPod came along and revolutionised the personal music player market).

Projects like Skype, open-source software and hyperscaled services have made fundamental changes to the telco industry. But what about OSS/BSS and the telcos themselves? Have we collectively innovated beyond the technical realm? Has there been any innovation from within that has re-framed the entire market?

As mentioned in an earlier post, I’ve recently been co-opted onto the TM Forum’s Transformation User Group to prepare a transformation framework and transformation guides for our industry. We’re creating a recipe book to help others to manage their OSS/BSS/telco/digital transformation projects. However, it only dawned on me over the weekend that I’d overlooked a really, really important consideration of any transformation – reframing!

Innovation needs to be a key part of any transformation, firstly because we need to consider how we’ll do things better. However, we also need to consider the future environment in which our transformed solutions will operate within. Our transformed solution will (hopefully) still be relevant and remain operational in 5, 10, 15 years from now. For it to be relevant that far into the future, it must be able to flexibly cater for the environmental situation in the future. Oh, and by the way, when I say “environment,” I’m not talking climate change, but other situational change. This might include factors like:

  • Networks under management
  • Delivery mechanisms
  • Business models
  • Information Technology platforms 
  • Architectural models
  • Process models
  • Staffing models (vs automations)
  • Geographical models (eg local vs global services)
  • Product offerings (driven by customer needs)
  • Design / Develop / Test / Release pipelines
  • Availability of funding for projects, and is it capex, opex, clip-the-ticket, etc
  • Risk models and risk adversity level
  • etc

Before embarking on each transformation project, we first need to challenge our current beliefs and assumptions to make sure they’re still valid now, but can remain valid into the next few years. Our current beliefs and assumptions are based on past experiences and may not be applicable for the to-be environment that our transformations will exist within.

So, how do we go about challenging our own beliefs and assumptions in this environment of massive, ongoing change?

Well, you may wish to ask “re-framing questions” to test your beliefs. These may be questions such as (but not limited to):

  1. Who are our customers (could be internal or external) and how do they perceive our products? Have we asked them recently? Have you spent much time with them?
  2. What is our supply chain and could it be improved? Are there any major steps or painful elements in the supply chain that could be modified or removed
  3. Are these products even needed under future scenarios / environments
  4. What does / doesn‘t move the needle? What even is “the needle”
  5. What functionality is visible vs invisible to customers 
  6. What data would be useful but is currently unattainable 
  7. Do we know how cash flows in our clients and our own companies in relation to these solutions? Specifically, how value is generated
  8. How easy are our products to use? How long does it take a typical user (internal / external) to reach proficiency 
  9. What personas DO we serve? What personas COULD we serve? What new personas WILL we serve
  10. What is the value we add that justifies our solution’s existence? Could that value be monetised in other ways
  11. Would alternative tech make a difference (voice recognition, AI, robotics, observability, biometric sensors, AR/VR, etc)
  12. Are there any strategic relationships that could transform this solution
  13. What does our team do better than anyone else (and what are they not as strong at)
  14. What know-how does the team possess that others don’t
  15. What features do we have that we just won’t need? Or that we absolutely MUST have
  16. Are there likely to be changes to the networks we manage
  17. Are there likely to be changes to the way we build and interconnect systems
  18. Where does customer service fall on the continuum between self-service and high-contact relationships
  19. What pricing model best facilitates optimal use of this solution (if applicable) (eg does a consumption-based usage / pricing model fit better than a capital investment model?). As Jeff Bezos says, “Your margin is my opportunity.” The incumbents have large costs that they need to feed (eg salaries, infrastructure, etc), whereas start-ups or volume-models allow for much smaller margins
  20. Where are the biggest risks of this transformation? Can they be eliminated, mitigated or transferred
  21. What aspects of the solution can be fixed and what must remain flexible
  22. What’s absorbing the most resources (time, money, people, etc) and could any of those resource-consumers be minimised, removed or managed differently

It also dawns on me when writing this list that we can apply these reframing questions not just to our transformation projects, but to ourselves – for our own personal, ongoing transformation.

I’d love to get your perspective below. What other re-framing questions should we ask? What re-framing exercise has fundamentally changed the way you think or approach OSS/BSS/personal transformation projects?

How to Approach OSS Vendor Selection Differently than Most

Selecting a new OSS / BSS product, vendor or integrator for your transformation project can be an arduous assignment.

Every network operator and every project has a unique set of needs. Counter to that, there are literally hundreds of vendors creating an even larger number of products to service those widely varied sets of needs.

If you’re a typical buyer, how many of those products are you already familiar with? Five? Ten? Fifty? How do you know whether the best-fit product or supplier is within the list you already know? Perhaps the best-fit is actually amongst the hundreds of other products and suppliers you’re not familiar with yet. How much time do you have to research each one and distill down to a short-list of possible candidates to service your specific needs? Where do you start? Lots of web searches?

Then how do you go about doing a deeper analysis to find the one that’s best fit for you out of those known products? The typical approach might follow a journey similar to the following:

The first step alone can take days, if not weeks, but also chews up valuable resources because many key stakeholders will be engaged in the requirement gathering process. The other downside of the requirements-first approach is that it becomes a wish-list that doesn’t always specify level of importance (eg “nice to have” versus “absolutely mandatory”).

Then, there’s no guarantee that any vendor will support every single one of the mandatory requirements. There’s always a level of compromise and haggling between stakeholders.

Next comes the RFP process, which can be even more arduous.

There has to be an easier way!

We think there is.

Our approach starts with the entire 400+ vendors in our OSS/BSS Vendor Directory. Then we apply two rounds of filters:

  1. Long-list Filter – Query by high-level product capability as per diagram below. For example, if you want outside plant management, then we filter by 9b, which returns a list of over 60 candidate vendors alone, but we can narrow it down further by applying filters 10 and 14 as well if you need these functionalities
  2. Short-list Filter – We then sit with your team to prepare a list of approx. 20 high-level questions (eg regions the vendor works in, what support levels they provide, high level functional questions, etc). We send this short questionnaire to the long-list of vendors for their clarification. Their collated responses usually then yields a short-list of 3-10 best-fit candidates that you/we can then perform a deeper evaluation on (how deep you dive depends on how thorough you want the review to be, which could include RFPs, PoCs and other steps).

The 2-step filter approach is arguably even quicker to prepare and more likely to identify the best-fit short-list solutions because it starts by assessing 400+ vendors, not just the small number that most clients are aware of.

The next step (step 5 in the diagram above) also uses a contrarian approach. Rather than evaluating via an RFP that centres around a set of requirements, we instead identify:

  • The personas (people or groups) that need to engage with the OSS/BSS
  • The highest-priority activities each persona needs to perform with the OSS/BSS
  • End-to-end workflows that blend these activities into a prioritised list of demonstration scenarios

These steps quickly prioritise what’s most important for the to-be solution to perform. We describe the demonstration scenarios to the short-listed vendors and ask them to demonstrate how their solutions solve for those scenarios (as best they can). The benefit of this approach is that the client can review each vendor demonstration through their own context (ie the E2E workflows / scenarios they’ve helped to design).

This approach does not provide an itemised list of requirement compliance like the typical approach. However, we’d argue that even the requirement-led approach will (almost?) never identify a product of perfect fit, even if it’s filled with “Will Comply” responses for functionality that requires specific customisation.

Our “filtering” approach will uncover the solution of closest fit (in out-of-the-box state) in a much more efficient way.

We should also highlight that the two chevron diagrams above are just sample vendor-selection flows. We actually customise them to each client’s specific requirements. For example, some clients require much more thorough analysis than others. Others have mandatory RFP and/or business case templates that need to be followed.

If you need help, either with:

  • Preparing a short-list of vendors for further evaluation down from our known list of 400+; or
  • Need help to perform a more thorough analysis and identify the best-fit solution/s

then we’d be delighted to assist. Please leave us a note in the contact form below.

New report – Inventory of the Future

I’ve recently been co-opted to lead the development of an “Inventory of the Future” transformation guide on behalf of TM Forum and thought you, the readers, might have some interesting feedback to contribute.

I had previously prepared an “Inventory of the Future” discussion paper prior to being invited into the TMF discussions about a month ago. However, my original pack had too much emphasis on a possible target state (so those slides have been omitted from the attachment below).

Click to access Inventory_of_the_Future_v9a.pdf

Instead, before attempting to define a target state under the guise of TM Forum, we first need to understand:

  1. Where carriers are currently at with their inventory solutions
  2. What they want to achieve next with their inventory and
  3. What will the future environment  (in say 5 years from now) look like that inventory will exist within. Eg
    1. Will it still integrate with orchestration (resource allocation), fault management (enrichment, SIA, RCA, etc), etc?
    2. Will it manage a vastly different type / style of network (such as massive networks of sensors / IoT that are managed as a cohort rather than individually)
    3. etc

This, dear reader, is where your opinions will be so valuable.

Do the objectives (slide 3) and problem statements (slide 4) resonate with you? Or do you have an entirely different perspective?

Would love to hear your thoughts in the comment section below!! Alternatively, if you’d prefer to share your ideas directly, please leave me a personal message. Leave a personal message if you’d like to see the rest of the report (as it currently stands) too.

New. Just Launched – Mastering Your OSS (the eBook version)

Featured

We’re excited to announce that we’ve just launched the eBook version of Mastering Your OSS.

Click on the link above or here to read all about it.

Do we need more dummies working on our OSS?

I was reading an article by Chris Campbell that states, “In his book How to Rule the World, Brian J. Ford reveals the relationship between sophistication and complexity; what he calls “obscurantism.” In short, the more sophisticated the problem-solver, the more complex (and costly) the solution.

Does that resonate with you in the world of OSS/BSS? Is it because we have so many unbelievably clever, sophisticated people working on them that they are so complex (and costly)??

Do we actually need more stupid, unsophisticated people (like me) to be asking the stupid questions and coming up with the simple solutions to problems?? Do we need to intentionally apply the dummy-lens??

Sadly, with so much mental horsepower spread across the industry, the dummy-lens is probably shouted down in most instances (in favour of the sophisticated / clever solutions… that turn out to be complex and costly). Check out this article about what impact complexity has on the triple constraint of project management.

I’d love to get your thoughts.

Even better than that, I’d love to hear any “dummy lens” ideas you have that must be considered right away!!

Drop us a note in the comment section below.

Time to Kill the Typical OSS Partnership Model?

A couple of years ago Mark Newman and the content team at TM Forum created a seminal article, “Time to Kill the RFP? Reinventing IT Procurement for the 2020s.” There were so many great ideas within the article. We shared a number of concordant as well as divergent ideas (see references #1, #2, #3, #4, #5, #6, and others).

As Mark’s article described, the traditional OSS/BSS vendor selection process is deeply flawed for both buyer and seller. They’re time-consuming and costly. But worst of all, they tend to set the project on a trajectory towards conflict and disillusionment. That’s the worst possible start for a relationship that will ideally last for a decade or more (OSS and BSS projects are “sticky” because they’re difficult to transform / replace once in-situ).

Partnership is the key word in this discussion – as reiterated in Mark’s report and our articles back then as well.

Clearly this message of long-held partnerships is starting to resonate, as we see via the close and extensive partnerships that some of the big service providers have formed with third-parties for integration and other services. 

That’s great…. but…… in many cases it introduces its own problem for the implementation of OSS and BSS projects. A situation that is also deeply flawed.

Many partnerships are based around a time and materials (T&M) model. In other words, the carrier pays the third-party a set amount per day for the number of days each third-party-provided resource works. A third-party supplies solution architects at ($x per day), business analysts at ($y per day), developers at ($z per day), project managers at… you get the picture. That sounds simple for all parties to wrap their head around and come to mutually agreeable terms on. It’s so simple to comprehend that most carriers almost default to asking external contractors for their daily charge-out rates.

This approach is deeply flawed – ethically conflicted even. You may ask why…. Well, Alan Weiss articulates it best as follows:

When you charge by the hour, you’re in ethical conflict with the client. You only receive larger pay for the longer you’re there, but the client is better served by how quickly you can meet objectives and leave.

Complex IT projects like OSS and BSS projects are the perfect example of this. If your partners are paid on a day rate, they’re financially incentivised for delays, bureaucracy, endless meetings and general inefficiency to prosper. In big organisations, these things tend to already thrive without any incentivisation!

Assuming a given project continues at a steady-state of resources, if a project goes twice as long as projected, then it also goes 100% over the client’s budget. By direct contrast, the third-party doubles their revenue on the project.

T&M partnership models disincentivise efficiency, yet efficiency is one of the primary reasons for the existence of OSS and BSS. They also disincentivise reusability. Why would a day-rater spend the extra time (in their own time) to systematise what they’ve learnt on a project when they know they will be paid by the hour to re-invent that same wheel on the next project?

Can you see why PAOSS only provides scope of work proposals (ie defined outcomes / deliverables / timelines and, most importantly, defined value) rather than day-rates (other than in exceptional circumstances)??

Let me cite just one example to illustrate the point (albeit a severe example of the point).

I was once assisting an OEM vendor to implement an OSS at a tier-1 carrier. This vendor also provided ongoing professional services support for tasks such as customisation. However, the vendor’s day-rates were slightly higher than the carrier was paying for similar roles (eg architects, developers, etc). The carrier invited a third-party to perform much of the customisation work because their day-rates were lower than the OEM.

Later on, I was tasked with reviewing a customisation written by the third-party because it wasn’t functioning as expected. On closer inspection, it had layers of nested function calls and lookups to custom tables in the OEM’s database (containing fixed values). It comprised around 1,500 lines of code. It must’ve taken weeks of effort to write, test and release into production via the change process that was in place. The sheer entanglement of the code took me hours to decipher. Once I finally grasped why it was failing and then interpreted the intent of what it should do, I took it back to a developer at the OEM. His response?

Oh, you’ve gotta be F#$%ing kidding me!

He then proceeded to replace the entire 1,500 lines and spurious lookup tables with half a line of code.

Let’s put that into an equation containing hypothetical numbers:

  • For the sake of the process, let’s assume test and release amounts are equivalent
  • OEM charges $1,000 per day for a dev
  • Third-party charges $900 per day for a dev
  • OEM developer (who knows how the OEM software works) takes 15 seconds to write the code  = $0.52
  • Third-party dev takes (conservatively) 5 days to write the equivalent code (which didn’t work properly) = $4,500

In the grand scheme of this multi-million dollar project, the additional $4,499.48 was almost meaningless, but it introduced weeks of delays (diagnosis, re-dev, re-test, re-release, etc).

Now, let’s say the new functionality offered by this code was worth $50,000 to the carrier in efficiency gains. Who deserves to be rewarded $5,000 for value delivered?

  • The OEM who completed the task and got it working in seconds (and was compensated $0.52); or
  • The Third-party who never got it to work despite a week of trying (and was compensated $4,500)

The hard part about scope of works projects is that someone has to scope them and define the value delivered by them. That’s a whole lot harder and provides less flexibility than just assigning a day rate. But perhaps that in itself provides value. If we need to consider the value of what we’re producing, we might just find that some of the tasks in our agile backlog aren’t really worth doing.

If you’d like to share your thoughts on this, please leave a comment below.

 

 

 

What Pottery and SpaceX Rockets Have in Common with OSS will Surprise You

In our previous article, we described five new OSS/BSS automation design rules applied by Elon Musk. Today, we’ll continue on into the second part of the Starbase video tour and lock onto another brilliant insight from Elon.

From 5:10 to 11:00 in the video below, Elon and Tim (of Everyday Astronaut) discuss why failure, in certain scenarios, is a viable option even in the world of rocket development.

An edited excerpt from the video is provided below.

[Elon] We have just a fundamentally different optimization for Starship versus say, like the polar extreme would be Dragon. Dragon, there can be no failures ever. Everything’s gotta be tested six ways to Sunday. There has to be tons of margin. There can never be a failure ever for any reason whatsoever.
Then Falcon is a little less conservative. It is possible for us to have, say, a failure of the booster on landing. That’s not the end of the world.
And then for Starship, it’s like the polar opposite of Dragon: we’re iterating rapidly in order to create the first ever fully reusable rocket, orbital rocket.
Anyway, it’s hard to iterate, though, when people are on every mission. You can’t just be blowing stuff up ’cause you’re gonna kill people. Starship does not have anyone on board so we can blow things up.

[Tim] Are you just hoping that by the time you put people on it, you’ve flown it say 100, 200 times, and you’re familiar with all the failure modes, and you’ve mitigated it to a high degree of confidence.

[Elon] Yeah.

Interesting…. Very interesting…

How does that relate to OSS? Well, first I’d like to share with you another story, this time about pottery, that I also found fascinating.

The ceramics teacher announced on opening day that he was dividing the class into two groups. All those on the left side of the studio, he said, would be graded solely on the quantity of work they produced, all those on the right solely on its quality. His procedure was simple: on the final day of class he would bring in his bathroom scales and weigh the work of the “quantity” group: fifty pounds of pots rated an “A,” forty pounds a “B,” and so on. Those being graded on “quality,” however, needed to produce only one pot—albeit a perfect one—to get an “A.” Well, come grading time and a curious fact emerged: the works of the highest quality were all produced by the group being graded for quantity. It seems that while the “quantity” group was busily churning out piles of work—and learning from their mistakes—the “quality” group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay
David Bayles and Ted Orland in their book, “Art and Fear: Observations on the Perils (And Rewards) of Art making,” which I discussed previously in this post years ago.

These two stories are all about being given the licence to iterate…. within an environment where there was also a licence to fail.

The consequences of failure for our OSS/BSS usually falls somewhere between these two examples. Not quite as catastrophic as crashing a rocket, but more impactful than creating some bad pottery.

But even within each of these, there are different scales of consequence. Elon presciently identifies that it’s less catastrophic to crash an unmanned rocket than it is to blow up a manned rocket, killing the occupants. That’s why he has different platforms. He can use his unmanned Starship platform to rapidly iterate, flying a lot more often than the manned Dragon platform by reducing compliance checks, redundancy and safety margins. 

So, let me ask you, what are our equivalents of Starship and Dragon?
Answer: Non-Prod and Prod environments!

We can set up non-production environments where it’s safe to crash the OSS/BSS rocket without killing anyone. We can reduce the compliance, run lots of tests / iterations and rapidly learn, identifying the risks and unknowns.

With OSS/BSS being software, we’re probably already doing this. Nothing particularly new in the paragraph above. But I’m less interested in ensuring the reliability of our OSS/BSS (although that’s important of course). I’m actually more interested in ensuring the reliability of the networks that our OSS/BSS manage.

What if we instead changed the lens to using the OSS/BSS to intentionally test / crash the (non-production or lab) network (and EMS, NMS, OSS/BSS too perhaps)? I previously discussed the concept of CT/IR – Continual Test / Incremental Resilience here, which is analogous to CI/CD (Continuous Integration / Continuous Delivery) in software. CT/IR is a method to automatically, systematically and programmatically test the resilience of the network, then using the learnings to harden the network and ensure resilience is continually improving. 

Like the SpaceX scenario, we can use the automated destructive testing approach to pump out high testing volumes and variants that could not occur if operating in risk-averse environments like Dragon / Production. Where planned, thoroughly tested changes to production may only be allowed during defined and pre-approved change windows, intentionally destructive tests can be run day and night on non-prod environments.

We could even pre-seed AIOps data sets with all the boundary failure cases ready for introduction into production environments without them having ever even been seen in the prod environment.

 

Zooming in and out of your OSS

Our previous post talked about using the following frame of reference to think bigger about our OSS/BSS projects and their possibilities.

It’s the multitudes of experts at Level 1 that get projects done and products released. All hail the doers!!

As Rory Sutherland indicated in his book, Alchemy – The surprising power of ideas that don’t make sense, “No one complained that Darwin was being trivial in comparing the beaks of finches from one island to another because his ultimate inferences were so interesting.”

In the world of OSS/BSS, we do need people who are comparing the beaks of finches. We need to zoom down into the details, to understand the data. 

But if you’re planning an OSS/BSS project or product; leading a team; consulting; or marketing / selling an OSS/BSS product or service, you also need to zoom out. You need to be like Darwin to look for, and comprehend the ramifications of, the details.

This is why I use WBS to break down almost every OSS/BSS project I work on. I start with the problem statements at levels 2-5 of the reference framework above (depending on the project) and try to take in the broadest view of the project. I then start breaking down the work at the highest level of granularity. From there, we can zoom in as far into the details as we need to. It provides a plan on a page that all stakeholders can readily zoom out of and back into, seeing where they fit in the overall scheme of the project.

With this same perspective of zooming in and out, I often refer to the solution architecture analogy. SAs tend to create designs for an end-state – the ideal solution that will appear at the end of a project or product. Having implemented many OSS/BSS, I’ve also seen examples of where the end-state simply isn’t achievable because of the complexity of all the moving parts (The Chessboard Analogy). The SAs haven’t considered all the intermediate states that the delivery team needs to step through or the constraints that pop up during OSS/BSS transformations. Their designs haven’t considered the detailed challenges along the way.

Interestingly, those challenges often have little to do with the OSS/BSS you’re implementing. It could be things like:

  • A lack of a suitable environment for dev, test, staging, etc purposes
  • A lack of non-PROD infrastructure to test with, leading to the challenging carve-out and protection of PROD whilst still conducting integration testing
  • A new OSS/BSS introduces changes in security or shared services (eg DNS, UAM, LDAP, etc) models that need to be adapted on the fly before the OSS/BSS can function
  • Carefully disentangling parts of an existing OSS/BSS before stitching in elements of your new functionality (The Strangler Fig transformation model)
  • In-flight project changes that are moving the end-state or need to be progressively implemented in lock-step with your OSS/BSS phased release, which is especially common across integrations and infrastructure
  • Changes in underlying platforms or libraries that your code-base depends on
  • Refactoring of other products like microservices 
  • The complex nuances of organisational change management (since our OSS/BSS often trigger significant change events)
  • Changes in market landscape
  • The many other dependencies that may or may not be foreseeable at the start of the journey

You need to be able to zoom out to consider and include these types of adjacency when planning an OSS/BSS.

I’ve only seen one person successfully manage an OSS/BSS project using bottom-up work breakdown (although he was an absolute genius and it did take him 14 hour days x 7 day weeks throughout the project to stay on top of his 10,000+ row Gantt chart and all of the moving dependencies within it). I’ve also seen other bottom-up thinkers nearing mental breakdown trying to keep their highly complex OSS/BSS projects under control.

Being able to zoom up and down may be your only hope for maintaining sanity in this OSS/BSS world (although it might already be too late for me…. and you???)

If you need guidance with the breakdown of work on your OSS/BSS project or need to reconsider the approach to a project that’s veering off target, give us a call. We’d be happy to discuss.

OSS Sandpit – Radio Planning Exercise

Wireless or radio technologies are making waves (sorry for the awful pun) at the moment. The telco world is abuzz with the changes being brought about by 5G, IoT, LEO satellite and other related tech. They all rely on radio frequency (RF) to carry their all-important communications information.

This article provides an example of preparing RF coverage maps by using the inventory module of our Personal OSS Sandpit Project as well as a freely available trial of Twinkler (www.twinkler.app). We’ve already shown some RF propagation in previous articles in this series, including:

But in today’s article, we’ll show radio coverage planning across the three sectors of a cellular network site.

This RF planning capability is becoming more widely useful with the advent of these technologies and the new business models / use-cases they support. Whereas RF planning used to be the domain of mobile network operators, we’re now seeing more widespread use including:

  • Neutral-host infrastructure owners
  • Wireless ISPs
  • IoT networks
  • Private mobile networks, especially for in-building or in-campus in-fill coverage (BTW, the Twinkler app allows both indoor and outdoor RF planning use-cases)
  • Government radio networks (eg emergency services)
  • Utilities
  • Enterprise (eg mining, transport / logistics, agriculture, broadcast radio / TV, etc)
  • Consulting services
  • Managed service offerings

In this sand-pit demo, we’ll reverse-engineer an operator’s existing tower / assets, but the same approach can be used to design and configure new assets (eg adding antenna for millimeter wave 5G). Here in Australia, ACMA provides a public record of all licenced comms transmitter / receiver devices. This link is an example of one site recorded in the ACMA database that we’ll use for this demo – https://web.acma.gov.au/rrl/site_search.site_lookup?pSITE_ID=204667 (Warrimoo Tower).

You may recall that we’d recently demonstrated building a 3D model of Warrimoo Tower. This was done by stitching together photos taken from a drone. Looking at the animated GIF below, you’ll notice that we’ve not just built a model of the tower, but also tagged the assets (antenna, combiners, radio units) attached to the tower in 3D space. If you look closely, you’ll notice the labels that appear at the end of the visual loop, which identify the names of each asset.

Whilst Warrimoo tower holds assets of many owners (different colours on the animation), we’ll focus specifically on one 3-sector cluster. We’ll specifically focus on 3 antennae owned by Optus transmitting in the 700MHz band (763 MHz centre frequency). These are highlighted in green in the 3D model above.

The steps we use for RF planning are as follows:

  1. Extract ACMA data into Kuwaiba, our inventory database
  2. Push data from Kuwaiba to Twinkler, our RF modelling tool.
  3. Visualise the radio coverage map

Step 1 – Tower / antenna data in inventory:

The following diagram shows the inventory of the tower in topological format. The 3-sector cluster we’re modelling has been circled in yellow. [Click on the image for a closer look]

You’ll also notice that we’ve specifically highlighted one antenna (in blue, which has the name “204667-ANT-81193-MG1-01 (Optus)” according to our naming convention). There’s a corresponding list of attributes in the right-hand pane relating to this antenna. Most of these attributes have been taken from the ACMA database, but could equally come from your OSS, NMS, asset management system or other network data sets.

Some of the most important attributes (for RF calculation purposes anyway) are:

  • Device make/model (as this defines the radiation profile)
  • Height (either above the base of the tower or sea-level, but above the base in our case)
  • Azimuth (direction the antenna is pointing)
  • Emission centre frequency (ie the frequency of transmission)
  • Transmission power

Step 2 – Twinkler takes these attributes and builds an RF model/s

You can either use the Twinkler application (sign up to a free account here – https://twinkler.io). The application can visualise coverage maps, but these can also be gathered via the Twinkler API if you wish to add them as an overlay in your inventory, GIS or similar tools (API details can be found here: https://twinkler.io/TwinklerAPITestClient.html).

Step 3 – Visualise radio coverage diagrams

As you can see from the attributes in the inventory diagram above, we have an azimuth of 230 degrees for the first antenna in the 3-sector group. The azimuths of the other two antennae are 15 and 140 degrees respectively.

These give us the following radiation patterns (each is a separate map, but I’ve combined to make an animated GIF for easier viewing):

You’ll notice that the combined spread in the diagram is slightly larger because the combined coverage is set to -130dBm signal level whereas the per-sector coverages are to -120dBm.

Note: You may have spotted that the mapping algorithm considers the terrain. From this zoomed-in view you’ll see the coverage black-spot in the gully in the top-left corner more easily.

Summary

I hope you enjoyed this brief introduction into how we’ve created a radio coverage map of an existing cellular network using the Inventory module of our Personal OSS Sandpit Project. Click on the link to step back to the parent page and see what other modules and/or use-cases are available for review.

 

Are you kidding? We’ll never use open-source OSS/BSS!

Back in the days when I first started using OSS/BSS software tools, there was no way any respectable telco was going to use open-source software (the other oss, for which I’ll use lower-case in this article) in their OSS/BSS stacks. The arguments were plenty, and if we’re being honest, probably had a strong element of truth in many cases back then.

These arguments included:

  • Security – This is the most commonly cited aversion I’ve heard to open-source. Our OSS/BSS control our network, so they absolutely have to be secure. Secure across all aspects of the stack from network / infrastructure to data (at rest and in motion) to account access to applications / code, etc. The argument against open-source is that the code is open to anyone to view, so vulnerabilities can be identified by hackers. Another argument is that community contributors could intentionally inject vulnerabilities that aren’t spotted by the rest of the community
  • Quality – There is a perception that open-source projects are more hobby projects  than professional. Related to that, hobbyists can’t expend enough effort to make the solution as feature-rich and/or user-friendly as commercial software
  • Flexibility – Large telcos tend to want to steer the products to their own unique needs via a lot of customisations. OSS/BSS transformation projects tend to be large enough to encourage proprietary software vendors to be paid to make the requested changes. Choosing open-source implies accepting the product (and its roadmap) is defined by its developer community unless you wish to develop your own updates
  • Support – Telcos run 24x7x365, so they often expect their OSS/BSS vendors to provide round-the-clock support as well. There’s a  belief that open-source comes with a best-effort support model with no contracted service obligations. And if something does go drastically wrong, that open-source disclaims all responsibility and liability
  • Continuity – Telcos not only run 24x7x365, but also expect to maintain this cadence for decades to come. They need to know that they can rely on their OSS/BSS today but also expect a roadmap of updates into the future. They can’t become dependent upon a hobbyist or community that decides they don’t want to develop their open-source project anymore

Luckily, these perceptions around open-source have changed in telco circles in recent years. The success of open-source organisations like Red Hat (acquired by IBM for $34 billion on annual revenues of $3.4 billion) have shown that valuable business models can be underpinned by open-source. There are many examples of open-source OSS/BSS projects driving valuable business models and associated professionalism. The change in perception has possibly also been driven by shifts in application architectures, from monolithic OSS/BSS to more modular ones. Having smaller modules has opened the door to utilisation of building block solutions like the Apache projects.

So let’s look at the same five factors above again, but through the lens of the pros rather than the cons.

  • Security – There’s no doubt that security is always a challenge, regardless of being open-source or proprietary software, especially for an industry like OSS/BSS where all organisations are still investing more heavily in innovation (new features/capabilitys) more than security optimisations. Clearly the openness of code means vulnerabilities are more easily spotted in open-source than in “walled-garden” proprietary solutions. Not just by nefarious actors, but its development community as well. Linus’ Law suggests that “given enough eyeballs, all bugs (and security flaws) are shallow.” The question for open-source OSS/BSS is whether there are actually many eyeballs. All commercially successful open-source OSS/BSS vendors that I’m aware of have their own teams of professional developers who control every change to the code base, even on the rare occasions when there are community contributions. Similarly, many modern open-source OSS/BSS leverage other open-source modules that do have many eyes (eg linux, snmp libaries, Apache projects, etc). Another common perception is security through obscurity, that there are almost no external “eyeballs.” The fragmented nature of the OSS/BSS industry means that some proprietary tools have a tiny install base. This can lull some into a false sense of security. Alternatively, open-source OSS/BSS manufacturers know there’s a customer focus on security and have to mitigate this concern. The other interesting perspective of openness is that open-source projects can quite quickly be scrutinised for security-code-quality. An auditor has free reign to identify whether the code is professional and secure. With proprietary software, the customer’s auditor isn’t afforded the same luxury unless special access is granted to the code. With no code access, the auditor has to reverse-engineer for vulnerabilities rather than foresee them in the code.
  • Quality – There’s no doubt that many open-source OSS/BSS have matured and found valuable business models to sustain them. With the profitable business model has come increased resources, professionalism and quality. With the increased modularity of modern architectures, open-source OSS/BSS projects are able to perform very specific niche functionalities. Contrast this with the monolithic proprietary solutions that have needed to spread their resources thinner across a much wider functional estate. Also successful open-source OSS/BSS organisations tend to focus on product development and product-related services (eg support), whereas the largest OSS/BSS firms tend to derive a much larger percentage of revenues from value-added services (eg transformations, customisations, consultancy, managed services, etc). The latter are more services-oriented companies than product companies. As inferred in the security point above, open-source also provides transparency relating to code-quality. A code auditor will quickly identify whether open-source code is of good quality, whereas proprietary software quality is hidden inside the black-box 
  • Flexibility – There has been a significant shift in telco mindsets in recent years, from an off-the-shelf to a build-your-own OSS/BSS stack. Telcos like AT&T have seen the achievements of the hyperscalers, observed the increased virtualisation of networks and realised they needed to have more in-house software development skills. Having in-house developers and access to the code-base of open-source means that telcos have (almost) complete control over their OSS/BSS destinies. They don’t need to wait for proprietary vendors to acknowledge, quote, develop and release new feature requests. They no longer rely on the vendor’s roadmap. They can just slip the required changes into their CI/CD pipeline and prioritise according to resource availability. Or if you don’t want to build a team of developers specifically skilled with your OSS/BSS, you can pick and choose – what functionality to develop in-house, versus what functionality you want to sponsor the open-source vendor to build
  • Support – Remember when I mentioned above that OSS/BSS organisations have found ways to build profitable business models around open-source software? In most cases, their revenues are derived from annual support contracts. The quality and coverage of their support (and the products that back it up) is directly tied to their income stream, so there’s commensurate professionalism assigned to support. As mentioned earlier, almost all the open-source OSS/BSS I’m aware of are developed by an organisation that controls all code change, not community consensus projects. This is a good thing when it comes to support, as the support partner is clear, unlike community-led open-source projects. Another support-related perspective is in the number of non-production environments that can be used for support, test, training, security analysis, etc. The cost of proprietary software means that it can be prohibitive to stand up additional environments for the many support use-cases. Apart from the underlying hardware costs and deployment effort, standing up additional open-source environments tends to be at no additional cost. Open-source also gives you greater choice in deciding whether to self-support your OSS/BSS in future (if you feel that your internal team knows the solution so intimately that they can fix any problem or functionality requirement that arises) rather than paying ongoing support contracts. Can the same be said for proprietary product support?
  • Continuity – This is perhaps the most interesting one for me. There is the assumption that big, commercial software vendors are more reliable than open-source vendors. This may (or might not) be the case. Plenty of commercial vendors have gone out of business, just as plenty of open-source projects have burned out or dwindled away. To counter the first risk, telcos pay to enter into software escrow agreements with proprietary vendors to ensure critical fixes and roadmap can continue even in the event that a vendor ceases to operate. But the escrow contract may not cover when a commercial vendor chooses to obsolete a product line of software or just fail to invest in new features or patches. This is a common event from even the world’s largest OSS/BSS providers. Under escrow arrangements, customers are effectively paying an insurance fee to have access to the code for organisational continuity purposes, not product continuity. Escrow may not cover that, but open-source is available under any scenario. The more important continuity consideration is the data and data is the reason OSS/BSS exist. When choosing a commercial provider, especially a cloud software / service provider, the data goes into a black box. What happens to the data inside the black box is proprietary and often what comes out of it is also. Telcos will tend to have far more control of their data destinies for operational continuity if using open-source solutions. Speaking of cloud and SaaS-model and/or subscription-model OSS/BSS, customers are at the whim of the vendor’s product direction. Products, features and modules can be easily changed or deprecated by these vendors, with little recourse for the buyers. This can still happen in open-source and become a impediment for buyers too, but at least open-source provides buyers with the opportunity to control their own OSS/BSS destinies.

Now, I’m not advocating one or the other for your particular situation. As cited above, there are clearly pros and cons for each approach as well as different products of best-fit for different operators. However, open-source can no longer be as summarily dismissed as it was when I first started on my OSS/BSS journey. There are many fine OSS and BSS products and vendors in our Blue Book OSS/BSS Vendor Directory that are worthy of your consideration too when looking into your next product or transformation.

Edit: Thanks to Guy B. who rightly suggested that Scalability was another argument against open-source in the past. Ironically, open-source has been a significant influencer in the almost unlimited scalability that many of our solutions enjoy today.

How to calculate the right jeopardy metrics in your end-to-end workflows

Last week we created an article that described how to use your OSS/BSS log data to generate reliable / quantifiable process flow diagrams.

We’ve expanded upon this research to identify a reliable calculation of jeopardy metrics. Jeopardy Management is the method for notifying operators when an in-flight workflow (eg customer order, etc) is likely to breach targets such as RFS date (ie when the customer’s service will be available for use) and/or SLAs (service level agreements) are likely to be breached.

Jeopardy management techniques are used to predict forward before a breach has occurred, hopefully. For example if an Order to Activate workflow for a particular product type consists of 10 steps and only the first 2 steps are completed within 29 days of a target of 30 days RFS, then we could expect that the RFS date is likely to be missed. The customer should be alerted. If the right trackers were built, this order should’ve had a jeopardy notification long before 29 days had elapsed. 

In the past, jeopardy indicators have tended to be estimated thresholds. Operators have tended to set notifications based on gut-feel (eg step 2 must be completed by day 5).  But through the use of log data, we can now provide a more specific jeopardy indicator for every step in the process.

The chart above shows every activity within a workflow across the horizontal axis. The vertical axis shows the number of days elapsed since the start of the workflow.

By looking at all past instances of this workflow, we can show the jeopardy indicator as a series of yellow dots. In other words, if any activity has ever been finished later than its corresponding yellow dot, then the E2E workflow it was part of has breached its SLA. 

To use a more familiar analogy it’s the latest possible date that you can study for exams and still be able to pass the subject, using time-stamped historical data. Not that I ever left it late to study for exams back in uni days!!  🙂

And yet if you look closely, you’ll notice that some blue dots (average elapsed time for this activity) in this example are higher than the jeopardy indicator. You’ll also notice that the orange dots (the longest elapsed time to complete this task across all instances of this workflow according to log data) are almost all above the jeopardy indicator. Those examples highlight significant RFS / SLA breaches in this data set (over 10% are in breach).

Leave us a note below if you’d like us to assist with capturing your jeopardy indicators and identifying whether process interventions are required across your OSS/BSS.

 

How to Document, Benchmark and Optimise Operational Processes

Have you been tasked with:

  1. Capturing as-is process flows (eg swim-lane charts or BPMN [Business Process Model and Notation] diagrams)
  2. Starting a new project where understanding the current state is important
  3. Finding ways to optimise day-to-day activities performed by your team
  4. Creating a baseline process to identify automation opportunities
  5. Comparing your current processes with recommendations such as eTOM or ITIL
  6. Identifying which tasks are leading to SLA / OLA breaches

As you may’ve experienced during project kick-off phases, as-is processes are usually not well defined, captured or adequately quantified (eg transaction volumes, duration times, fall-outs, etc) by many customers. 

If process diagrams have been captured, they’re often theoretical workflow maps developed by Business Analysts and Subject Matter Experts to the best of their knowledge. As such, they don’t always reflect real and/or complete flows. They may have no awareness of the rare flows / tasks / situations that can often trip our OSS/BSS tools and operators up. The rarer the sub-flows, the less likely they are to be documented.

Even if the flows have been fully documented, real metrics / benchmarks are rarely recorded. Metrics such as end-to-end completion times and times taken between each activity within the flow can be really challenging to capture and visualise, especially when you have large numbers of flows underway at any point in time.

Do you struggle to know where the real bottlenecks are in your process flows? Which tasks cause fall-outs? Which team members need advanced training? Which process steps have the largest differences in max / min / average durations? Which steps are justified to build automations for? As the old saying goes, if you can’t measure it, you can’t manage it.

You need quantitative, not qualitative understanding of your workflows

As a result, we’ve developed a technique to reverse-engineer log data to map and quantify processes. Logs that our OSS/BSS routinely collect and automatically time-stamp. By using time-stamped logs, we can trace every step, every flow variant, every sequence in the chain and every duration between them. This technique can be used on fulfilment, assurance and other flows. The sample below shows transaction volumes / sequences, but can also show durations within the flows:

Note that this and subsequent diagrams have been intentionally left in low-res format here on this page.

Better than just volumes, we can compare the max / mean / min processing times to identify the duration of activities and show bottlenecks (thicker red lines in the diagram below) as well as identifying hold-ups and inconsistencies in processing times:

By combining insights from flow volumes and timings, we can also recommend the processes and/or steps that optimisation / automations are most justified for.

We can also use monitoring of the flows to identify failure situations that have occurred with a given process, such as the examples highlighted in red below. 

We can also use various visualisation techniques to identify changes / trends in processing over time. These techniques can even assist in identifying whether interventions (eg process improvements or automations) are having the intended impacts.

The following chart (which is clickable) can be used to identify which tasks are consistently leading to SLA (Service Level Agreement) breaches.

The yellow dots indicate the maximum elapsed time (from the start of a given workflow) that has not resulted in the SLA breach. In other words, if this activity has ever been finished later than the yellow dot, then the E2E workflow it was part of has breached its SLA. These numbers can be used during a workflow to predict likelihood that it will breach SLA. It can also be used for setting jeopardy values to notify operators of workflow slippages.

There are a few other points of interest in this chart:

  • Orange dots indicate the longest elapsed time for this activity seen within all flows in the log data
  • Grey dots indicate the shortest elapsed time from the beginning of a workflow
  • Blue dots indicate the average elapsed time
  • Yellow dots are the Jeopardy indicator, meaning that if the elapsed time of this activity has ever exceeded this value then it has gone on to breach SLA
  • The red line is SLA threshold for this particular workflow type
  • Shaded box shows tasks that have never been in an E2E flow that has met SLA
  • You’ll notice that many average values (blue dots) are above jeopardy, which indicates this activity is regularly appearing in flows that go on to breach SLA levels
  • Almost all max values are above jeopardy (most are so high that they’re off the top of the scale) so most activities have been part of an E2E flow that has breached SLA
  • The shaded blue box shows tasks that have never been in an E2E flow that has met SLA!!
  • Needless to say, there were some interventions required with this example!

Operational Process Summary

As described above, using log data that you probably already have ready access to in your OSS/BSS, we can assist you to develop quantifiable process flow information. Having quantifiable data in turn can lead to greater confidence in initiating process interventions, whether they are people (eg advanced training), process (eg business process re-engineering) or technology (eg automations).

This technique works equally well for:

  • Understanding the current situation before commencing an OSS/BSS transformation project
  • Benchmarking and refining processes on an OSS/BSS stack that is running in business-as-usual mode
  • Highlighting the impact a process intervention has had (ie comparing before and after)

Would you like to book a free consultation to discuss the challenges you face with your (or a customer’s) as-is process situation? Please leave your details and list of challenges in the contact form below.

How to Transform your OSS/BSS with Open APIs

The video below, starring Martin Pittard, the Principal IT Architect at Vocus Group, provides a number of important OSS/BSS Transformation call-outs that we’ll dive into in this article.

Vocus has embarked on a journey to significantly overhaul its OSS/BSS stack and has heavily leveraged TM Forum’s Open API suite to do so. One of the primary objectives of the overhaul was to provide Vocus with business agility via its IT and digital assets.

As Martin indicates, all transformation journeys start with the customer, but there are a few other really interesting call-outs to make in relation to this slide below:

  • The journey started with 6 legacy networks and 8 legacy BSS stacks. This is a common situation facing many telcos. Legacy / clutter has developed over the years. It results in what I refer to as The Chessboard Analogy, which precludes business agility. Without a significant transformation, this clutter constrains you to incremental modifications, which leads to a Strangulation of Feature Releases.
  • Over 60,000 SKUs (Stock-keeping Units or distinct product offerings) and 100+ order characteristics is also sadly not unusual. The complexity ramifications of having this many SKUs is significant. I refer to it as The Colour Palette Analogy. The complexity is not just in IT systems, but also for customers and internal teams (eg sales, contact centre, operations, etc)
  • No end-to-end view of inventory. AKA Information of all sorts, not just inventory, spread across many siloes. The ramifications of this are also many, but ultimately it impacts speed of insight (and possibly related SLA implications). Whether that’s via swivel-chairing between products, data scientists having to determine ways to join disparate (and possibly incompatible) data sets, data quality / consistency implications, manual workarounds, etc
  • Manual Sales Processes and “Price on Availability” tend to also imply a lack of repeatability / consistency across sale (and other) processes. More repeatability means more precision, more room for continual improvement and more opportunity to automate and apply algorithmic actions (eg AI/ML-led self-healing). It’s the Mona Lisa of OSS. Or to quote Karl Popper, “Non-reproducible single occurrences are of no significance to science”… or OSS/BSS.

The Vocus transformation was built upon the following Architectural Principles:

Interestingly, Martin describes that Vocus has blended TM Forum SID and MEF data models in preparing its common data model, using standards of best-fit.

Now, this is the diagram that I most want to bring to your attention.

It shows how Vocus has leveraged TM Forum’s Open APIs across OSS/BSS functionalities / workflows in its stack. It shows actual Open API numbers (eg TMF620). You can find a table of all available Open APIs here including specifications, Swagger, Postman, etc resources.

A key to the business agility objective is making product offerings catalog-driven. Vocus’ broad list of product offerings are described as:

And underpinning the offerings are a hierarchical, catalog of building blocks:

Martin also raises the importance of tying operational processes / data-flows like telemetry, alarms, fault-fix, etc to the CFS entities, referring to it as, “Having a small set of tightly bounded reusable components.”

Martin then provides a helpful example to show flows between actual suppliers within their transformed stack, including the Open API reference number. In this case, he shows the CPQ (Configure, Price, Quote) process as part of the L2Q (Lead to Quote) process:

Note that the continuation of decomposition of actions is also described in the video, but not shown in this article.

And finally, Martin outlines the key learnings during the Vocus OSS/BSS transformation:

Final call-outs from Martin include:

  • By following standards as much as possible means less unique tech-debt to maintain
  • The dynamic architecture was the key to this transformation
  • The Open APIs have varying levels of maturity and vendor support. These Open APIs are definitely a work in progress, with evolving contributions from TM Forum members
  • Partnering with organisations that have experience with these technologies are important, but even more important is retaining design and architecture accountability within Vocus and similar carriers
  • Careful consideration of whether data needed to be retained or orphaned when bringing across to the new EIM (Enterprise Information Model)
  • There remains swivel-chair processes on some of the smaller-volume flows that didn’t warrant an API
  • Federated customer, network and services inventory information will continue to inter-work with legacy networks

Martin has also been kind enough to be a guest on The Passionate About OSS Podcast (Episode 019), where he provides additional layers of discussion to this article.

And a final call-out from me. It’s a really detailed and generous sharing of information about the recent OSS/BSS Transformation by Vocus and Martin. It’s well worth taking a closer look and listen for anyone embarking on an OSS/BSS transformation.

How to Avoid the Pitfalls of OSS Sharecropping

Firstly, what is Digital Sharecropping? Nicholas Carr coined the term Digital Sharecropping all the way back in 2006, citing the phenomenon where, “One of the fundamental economic characteristics of Web 2.0 is the distribution of production into the hands of the many and the concentration of the economic rewards into the hands of the few.” In other words, the landholder or platform owner (eg Facebook) is able to derive the majority of the financial benefits of the toil of the platform users (eg content creators). In isolation, the platform users don’t deliver much value, but they do in aggregate. Yet the platform owner potentially derives the majority of the aggregated commercial benefit.

But you’re probably wondering what Digital Sharecropping has to do with OSS/BSS. First let me ask you the question:

Who are the Landholders / Platform Owners of OSS and BSS Industries?

Let’s answer that question by taking a Dickensian look at OSS of past, present and future.

OSS Past: The OSS and BSS of the past were often built using third-party tools such as Oracle Forms, AutoCAD, CORBA, SNMP collection libraries, various GIS, programming languages such as C++, etc.

OSS Present: The OSS, and particularly the BSS, of today are increasingly being built on cloud environments, Salesforce and similar.

OSS Future: In future, we’re likely to use even more of the hyperscaler tools, leveraging their pre-packaged AI, mixed reality, data ETL, visualisation and other tools. The benefits of SaaS and subscription revenue models to software providers means that there’s a likelihood that many more platform plays will be built for OSS vendors to use in future.

What are the Benefits of using Third-Party Platforms?

Put simply, using third-party tools can be a very effective way for OSS vendors to deliver products to market faster and cheaper. The reality is that most OSS companies don’t have the time or requisite skills to be able to create some of these tools (eg AI). OSS vendors  get to stand on the shoulders of giants!

So what’s all the fuss I’m making about this then? The answer can be quickly summarised in the phrase – Locus of Control.

The Pitfalls of Relying on Third-Party Platforms

The dangers of being heavily reliant on specific third-party platforms are varied and numerous. The following factors apply to OSS/BSS vendors and their network operator customers alike:

  • Roadmap Alignment – when you first choose to align with a landowner (third-party product / platform), first you decide on the product of best alignment, then you design your own product to fit within the constraints of all that it offers. However, over time the third-party is making their own roadmap decisions. Unless you’re a keystone customer of theirs, they’re probably making architectural decisions that have no alignment with where you want to take your own product
  • Refactoring – the landowner is probably making their own enhancements to their platform. In some cases, those changes mean your OSS also needs resources assigned to be refactored, taking away from the resources required to make enhancements to your own product
  • Price changes – the landowner can choose to make changes to their pricing structure at any time. What might have been a profitable long-term contract between OSS vendor and customer can easily become loss-making if the landowner decides to raise their fees, either through direct price increases or restructuring the cost model. If your product is tightly coupled to theirs, you don’t have the option of quickly transitioning to an alternate supplier. As Nicholas also described, “the sharecroppers operate happily in an attention economy while their overseers operate happily in a cash economy.” The platform players are in a position of power where they can strip out all the benefits of the cash economy, leaving OSS sharecroppers with only the attention economy (including the support of OSS products for their customers, the network operators)
  • Reputation (bug fix) – when you’re reliant on “black box” functionality supplied by others, where you’re unable to control the inner workings of that functionality, you don’t have end-to-end control over issues customers have with your OSS products. Your reputation is inextricably linked to the reliability, scalability, etc of the platform you rely upon
  • Change of Ownership (Ophaning) – if the landholder decides to sell, there’s the possibility that the platform you rely upon could be obsoleted, switched to their own alternative product or just left to languish (see next point)
  • Change in Support – landholder support comes in many different forms:
    • Functional Obsolescence – No additional functionality or roadmap is assigned
    • Professional Services – No professional services are provided to assist with ongoing development of your product
    • Patching – No patches or security fixes are supplied. This could also include an ability to support other third-party software dependencies (eg latest edition of Java)
    • Service Levels – Changes in support models and SLAs
  • Loss of Relevance – Some OSS today have a reliance on a choice of platform they made decades ago. Millions of developer hours have been invested into the product and that platform since. However, there can be structural shifts that mean the platform, technology or processes are no longer preferred practices. An example could be the CAD drawings (and A1 print-outs) that were used by field techs in the past have begun to be replaced by interactive applications viewed on smartphones or tablets. 

So, how to avoid the Pitfalls of Digital Sharecropping?

The benefits of leveraging third-party platforms are too useful to ignore. However, lock-in / dependence avoidance strategies should also be considered. The secret is to spend most of your effort building the assets that you have direct control over. For example, rather than building your code / logic within the platform’s tools, look to extricate that logic into your own modules, microservices, etc.

To follow the old adage, don’t build on rented land… which reminds me of a story a Sydney taxi driver once shared. He described in great detail how he’d invested $250,000 and years of his own effort building extensions to his home (right down to the placement of concrete gargoyles). However, my jaw hit the floor of his taxi when he finished his story with, “… I hope my landlord doesn’t mind!!” 

 

There’s an OSS Security Elephant in the Room!

The pandemic has been beneficial for the telco world in one way. For many who weren’t already aware, it’s now clear how incredibly important telecommunications providers are to our modern way of life. Not just for our ability to communicate with others, but our economy, the services we use, the products we buy and even more fundamentally, our safety.

Working in the telco industry, as I’m sure you do, you’ll also be well aware of all the rhetoric and politics around Chinese manufactured equipment (eg Huawei) being used in the networks of global telco providers. The theory is that having telecommunications infrastructure supplied by a third-party, particularly a third-party aligned with non-Western allies, puts national security interests at risk.

In this article, “5G: The outsourced elephant in the room,” Bert Hubert provides a brilliant look into the realities of telco network security that go far beyond just equipment supply. It breaks the national security threat into three key elements:

  • Spying (using compromised telco infrastructure to conduct espionage)
  • Availability (compromising and/or manipulating telco infrastructure so that it’s unable to work reliably)
  • Autonomy (being unable to operate a network or to recover from outages or compromises)

The first two are well understood and often discussed. The third is the real elephant in the room. The elephant OSS/BSS have a huge influence over (potentially). But we’ll get to that shortly.

Before we do, let’s summarise Bert’s analysis of security models. For 5G, he states that there’s an assumption that employees at national carriers design networks, buy equipment, install it, commission it and then hand it over to other employees to monitor and manage it. Oh, and to provide other specialised activities like lawful intercept, where a local legal system provides warrants to monitor the digital communications of (potentially) nefarious actors. Government bodies and taxpayers all assume the telcos have experienced staff with the expertise to provide all these services.

However, the reality is far different. Service providers have been outsourcing many of these functions for decades. New equipment is designed, deployed, configured, maintained and sometimes even financed by vendors for many global telcos. As Bert reinforces, “Just to let that sink in, Huawei (and their close partners) already run and directly operate the mobile telecommunication infrastructure for over 100 million European subscribers.

But let’s be very clear here. It’s not just Huawei and it’s not just Chinese manufacturers. Nor is it just mobile infrastructure. It’s also cloud providers and fixed-line networks. It’s also American manufacturers. It’s also the integrators that pull these networks and systems together. 

Bert also points out that CDRs (Call Detail Records) have been outsourced for decades. There’s a strong trend for billing providers to supply their tools via SaaS delivery models. And what are CDRs? Only metadata. Metadata that describes a subscriber’s activities and whereabouts. Data that’s powerful enough to be used to assist with criminal investigations (via lawful intercept). But where has CDR / bill processing been outsourced to? China and Israel mostly.

Now, let’s take a closer look at the autonomy factor, the real elephant in the room. Many design and operations activities have been offshored to jurisdictions where staff are more affordable. The telcos usually put clean-room facilities in place to ensure a level of security is applied to any data handled off-shore. They also put in place contractual protection mechanisms.

Those are moot points, but still not the key point here. As Bert brilliantly summarises,  “any worries about [offshore actors] being able to disrupt our communications through backdoors ignore the fact that all they’d need to do to disrupt our communications.. is to stop maintaining our networks for us!

There might be an implicit trust in “Western” manufacturers or integrators (eg Ericsson, Nokia, IBM) in designing, building and maintaining networks. However, these organisation also outsource / insource labour to international destinations where labour costs are cheaper.

If the R&D, design, configuration and operations roles are all outsourced, where do the telcos find the local resources with requisite skills to keep the network up in times when force majeure (eg war, epidemic, crime, strikes, etc) interrupts a remote workforce? How do local resources develop the required skills if the roles don’t exist locally?

Bert proposes that automation is an important part of the solution. He has a point. Many of the outsource arrangements are time and materials based contracts, so it’s in the resource suppliers’ best interests for activities to take a lot of time to maintain manually. He counters by showing how the hyperscalers (eg Google) have found ways of building automations so that their networks and infrastructure need minimal support crews.

Their support systems, unlike the legacy thinking of telco systems, have been designed with zero-touch / low-touch in mind.

If we do care about the stability, resiliency and privacy of our national networks, then something has to be done differently, vastly different! Having highly autonomous networks, OSS, BSS and related systems is a start. Having a highly skilled pool of local resources that can research, design, build, commission, operate and improve these systems would also seem important. If the business models of these telcos can’t support the higher costs of these local resources, then perhaps national security interests might have to subsidise these skills?

I wonder if the national carriers and/or local OSS/BSS / Automation suppliers are lobbying this point? I know a few governments have inserted security regulatory systems and pushed them onto the telcos to adhere to, to ensure they have suitable cyber-security mechanisms. They also have lawful intercept provisions. But do any have local operational autonomy provisions? None that I’m aware of, but feel free to leave us a comment about any you’re aware of.

PS. Hat tip to Jay for the link to Bert’s post.

Uses of OSS Augmented Reality in the Data Centre

I was doing some research on Ubiquiti’s NMS/OSS tools yesterday and stumbled upon the brilliant Augmented Reality (AR) capability that it has (I think).

I’ve converted to a video shown below (apologies for the low-res – you can try clicking here for Ubiquiti’s full-res view)

I especially love how it uses the OLED at the left side of the chassis almost like a QR code to uniquely identify the chassis and then cross-link with current status information pulled from Ubiquiti’s UNMS (assumedly).

As mentioned in this post from all the way back in 2014, the potential for this type of AR functionality is huge if / when linked to OSS / BSS data. Some of the potential use-cases for inside the data centre as cited in the 2014 article were:

  1. Tracing a cable (or patchlead) route through the complex cabling looms in a DC
  2. Showing an overlay of key information above each rack (and then each device inside the rack):
    1. Highest alarm severity within the rack (eg a flashing red beacon above each rack that has a critical alarm on any device within it, then a red beacon on any device inside the rack that has critical alarms)
    2. Operational state of each device / card / port within the rack
    3. Active alarms on each device
    4. Current port-state of each port
    5. Performance metrics relating to each device / port (either in current metric value or as a graph/s)
    6. Configuration parameters relating to each device / port (eg associated customer, service, service type or circuit name)
  3. Showing design / topology configurations for a piece of equipment
  4. Showing routing or far-end connectivity coming from a physical port (ie where the cable ultimately terminates, which could be in the same DC or in another state / country)

I believe that some of these features have since been implemented in working solutions or proofs-of-concept but I haven’t seen any out in the wild. Have you?

I’d love to hear from you if you’ve already used these Ubiquiti tools and/or have seen AR / OSS solutions actually being used in the field. What are your thoughts on their practicality?

What other use-cases can you think of? Note that the same 2014 article also discusses some AR use-cases that extend beyond the DC.

How To Optimise A Network Assurance GUI To Get Results

In the old-school world of network assurance, we just polled our network devices and aggregated all the events into an event list. But then our networks got bigger and too many events were landing in the list for our assurance teams to process.

The next fix was to apply filters. For example, that meant dropping the Info and Warning messages because they weren’t all that important anyway…. were they?

But still, the event list just kept scrolling off the bottom of the page. Ouch. So then we looked to apply correlation and suppression rules. That is, to apply a set of correlations so that some of the alarms could be bundled together into a single event, allowing the “child” events to be suppressed.

Then we can get a bit more advanced with our rules and perform root-cause analysis (RCA). Now, we’re moving to identify patterns using learning algorithms… to reduce the volume of the event list. But with virtualised networks, higher-speed telemetry and increased network complexity, the list keeps growing and the rules have to get more “dynamic.”

Each of these approaches is designing a more highly filtered lens through which a human operator can view the health of the network. The filters and rules are effectively dumbing down the information that’s landing with the operator to solve. The objective appears to be to develop a suitably dumbed-down solution that allows us to throw lots of minimally-trained (and cheaper) human operators at the (still) high transaction count problem. That means the GUI is design to filter out and dumb down too.

But here’s the thing. The alarm list harks back decades to when telcos were happy having a team of Engineers running the NOC, resolving lots of transactions. Fast forward to today and the telcos aspire to zero-touch assurance. That implies a solution that’s designed with machines in mind rather than humans. What we really want is for the network to self-heal based on all the telemetry it’s seeing.

Unfortunately, rare events can still happen. We still need human operators in the captain’s seat ready to respond when self-healing mechanisms are no longer able to self-heal.

So instead of dumbing-down and filtering out for a large number of dumbed-down and filtered out operators, perhaps we could consider doing the opposite.

Let’s continue to build networks and automations that take responsibility for the details of every transaction (even warning / info events). But let’s instead design a GUI that is used by a small number of highly trained operators, allowing them to see the overall network health posture and respond with dynamic tools and interactions. Preferably before the event using predictive techniques (that might just learn from all those warning / info events that we would’ve otherwise discarded).

Hat tip to Jay for some of the contrarian thoughts that seeded this post.

 

OSS Functionality – Is Your Focus In Anonymous Places?

Yesterday’s article asked whether OSS tend to be anonymous and poorly designed and then compared how Jony Ive (who led the design of iPads, iPods, iPhones for Apple) might look at OSS design. Jony has described “going deep” – being big on focus, care and detail when designing products. The article looked at 8 care factors, some of which OSS vendors do very well and others, well, perhaps less so.

Today we’ll un-pack this in more detail, using a long-tail diagram to help articulate some of the thoughts. [Yes, I love to use long-tail diagrams to help prioritise many facets of OSS]

The diagram below comes from an actual client’s functionality usage profile.
Long tail of OSS

The x-axis shows a list of functionalities / use-cases. The y-axis shows the number of uses (it could equally represent usefulness or value or other scaling factor of your choice).

The colour coding is:

  • Green – This functionality exists in the product today
  • Red – This functionality doesn’t exist

The key questions to ask of this long-tail graph are:

  1. What functionality is most important? What functionality “moves the needle” for the customers? But perhaps more importantly, what functionality is NOT important
  2. What new functionality should be developed
  3. What old functionality should be re-developed

Let’s dive deeper.

#1 – What functionality is most important
In most cases, the big-impact demands on the left side of the graph are going to be important. They’re the things that customers need most and use most. These are the functions that were included in the MVP (minimum viable product) when it was first released. They’ve been around for years. All the competitors’ products also have these features because they’re so fundamental to customers. But, because they already exist, many vendors rarely come back to re-factor them.

There are other functions that are also important to customers, but might be used infrequently (eg data importers or bulk processing tools). These also “move the needle” for the customer.

#2 – What functionality should be developed (and what should not)
What I find interesting is that many vendors just add more and more functionality out at the far right side of the graph, adding to the hundreds of existing functions. They can then market all those extra features, rightly saying that their competitors don’t have these abilities…. But functionality at the far right rarely moves the needle, as described in more detail in this earlier post!

Figuring out what should be green (included) and what should be red (excluded) on this graph appears to be something Apple has done quite well with its products. OSS vendors… perhaps less so!! By the way, the less green, the less complexity (for users, developers, testers, integrators, etc), which is always an important factor for OSS implementations.

Yesterday’s post mentioned eight “care factors” to evaluate your OSS products / implementations by. But filtering out the right side of the long-tail (ie marking up more red boxes) might also help to articulate an important “don’t care factor.”

#3 – What functionality should be re-developed
Now this, for me, is the important question to ask. If the green boxes are most important, especially the ones at the far left of the graph, should these also be the ones that we keep coming back to, looking to improve them? Where usability, reliability, efficiency, de-cluttering, etc are most important?

I suspect that Apple develop hundreds of prototypes that focus on and care about the left side of the graph in incredible detail, whilst looking to reduce the green bars in the right side of the graph. My guess is that subsequent updates to their products also seek improvements to the left side…. whilst also adding some new features, turning some of the red boxes green, but rarely all the way out to the right edge of the graph. 

Summary Questions

If you are in any way responsible for OSS product development, where is your “heat map” of attention on this long tail? Trending towards the left, middle or right?

But, another question vexes me. Do current functionality-based selection / procurement practices in OSS perpetuate the need to tick hundreds of boxes (ie the right side of the long tail), even though many of those functions don’t have a material impact? There’s a reason I’ve moved to a more “prioritised” approach to vendor selection in recent years, but I suspect the functionality check-boxes of the past are still a driving force for many.

OSS – Are they anonymous, poorly made objects?

We’re surrounded by anonymous, poorly made objects. It’s tempting to think its because the people who use them don’t care – just like the people who make them. But what [Apple has] shown is that people do care. It’s not just about aesthetics. They care about things that are thoughtfully conceived and well made.”
Jony Ive (referenced here).

As you undoubtedly know, Jony Ive is the industrial design genius behind many of Apple’s ground-breaking products like the iPod, iPad, etc. You could say he knows a bit about designing products.

I’d love to see what he would make of our OSS. I suspect that he’d claim they (and I’m using the term “they” very generically here) are poorly made objects. But I’d also love the be a fly on the wall to watch how he’d go about re-designing them.

It’s doing an extreme disservice to say that all OSS are poorly designed. From a functionality perspective, they are works of art and incredible technical achievements. They are this MP3 player:

Brilliantly engineered to provide the user with a million features.

But when compared with Apple’s iPods????

The iPods actually have less functionality than the MP3 player above. But that’s part of what makes them incredibly intuitive and efficient to use.

Looking at the quote above from Ive, there’s one word that stands out – CARE. As product developers, owners, suppliers, integrators, do we care sufficiently about our products? This question could be a little incendiary to OSS product people, but there are different levels of care that I’ve seen taken in the build of OSS products:

  • Care to ensure the product can meet the requirements specified? Tick, yes absolutely in almost every case. Often requiring technical brilliance to meet the requirement/s
  • Care to ensure the code is optimised? In almost all cases, yes, tick. Developers tend to have a passionate attention for detail. Their code must not just work, but work with the most efficient algorithm possible. Check out this story about a dilemma of OSS optimisation
  • Care to ensure the user experience is optimised? Well, to be blunt, no. Many of our OSS are not intuitive enough. Even if they were intuitive once, they’ve had so much additional functionality bolted on, having a Frankenstein effect. Our products are designed and tested by people who are intimately familiar with our products’ workings. How often have you heard of products being tested by an external person, a layperson, or even a child to see how understandable they are? How often are product teams allowed the time to prototype many different UI design variants before releasing a new functionality? In most cases, it’s the first and only design that passed functional testing that’s shipped to customers. By contrast, do you think Apple allowed their first prototype of the iPod to be released to customers?
  • Care to ensure that bulk processing is optimised? In most cases, this is often also a fail. OSS exist to streamline operations, to support high volume transactions (eg fulfillment and assurance use-cases). But how many times have you seen user interfaces and test cases that are designed for a single transaction, not to benchmark the efficiency of transactions at massive scale?
  • Care to ensure the product can pass testing? Tick, yes, we submit our OSS to a barrage of tests, not to mention the creation of modern test automation suites
  • Care to ensure the product is subjected to context-sensitive testing? Not always. Check out this amusing story of a failure of testing.
  • Care to ensure that installation, integration and commissioning is simple and repeatable? This is an interesting one. Some OSS vendors know they’re providing tools to a self-service market and do a great job of ensuring their customers can become operational quickly. Others require an expert build / release team to be on site for weeks to commission a solution. Contrast this again with the iPad. It’s quick and easy to get the base solution (including Operating System and core apps) operational. It’s then equally easy to download additional functionality via the App Store. Admittedly, the iPad doesn’t need to integrate with legacy “apps” and interfaces in the same way that telco software needs to!! Eeeek!
  • Care about the customer? This can also be sporadic. Some product people / companies pay fastidious attention to the customers, the way they use the products / processes / data and the objectives they need to meet using the OSS. Others are happy to sit in their ivory towers, meet functional testing and throw the solution over the fence for the customers to deal with.

What other areas have I missed? In what other ways do we (or not) take the level of care and focus that Apple does? Leave us a comment below.

How to Design an OSS / Telco Data Governance Framework

Data governance constructs a strategy and a framework through which an organization (company as well as a state…) can recognize the value of its data assets (towards its own or global goals), implement a data management system that can best leverage it whilst putting in place controls, policies and standards that simultaneously protect data (regulation & laws), ensure its quality and consistency, and make it readily available to those who need it.”
TM Forum Data Governance Team.

I just noticed an article on TM Forum’s Inform Platform today entitled, “Telecoms needs a standardized, agile data governance framework,” by Dawn Bushaus. 

The Forum will publish a white paper on data governance later this month. It has been authored with participation from a huge number of companies including Antel Uruguay, ATC IP, BolgiaTen, Deloitte, Ernst & Young, Etisalat UAE, Fujitsu, Globe Telecom, Huawei Technologies, International Free and Open Source Solutions Foundation, International Software Techniques, KCOM Group, Liquid Telecom, Netcracker Technology, Nokia, Oracle, Orange, Orange Espagne, PT XL Axiata, Rogers Communications, stc, Tech Mahindra, Tecnotree, Telefonica Moviles, Telkom and Viettel. Wow! What a list of luminaries!! Can’t wait to read what’s in it. I’m sure I’ll need to re-visit this article after taking a look at the white paper.

It reminded me that I’ve been intending to write an article about data governance for way too long! We have many data quality improvement articles, but we haven’t outlined the steps to build a data governance policy.

One of my earliest forays into OSS was for a brand new carrier with a brand new network. No brownfields challenges (but plenty of greenfields challenges!!). I started as a network SME, but was later handed responsibility for the data migration project. Would’ve been really handy to have had a TM Forum data governance guide back then! But on the plus side, I had the chance to try, fail and refine, learning so much along the way.

Not least of those learnings was that every single other member on our team was dependent on the data I was injecting into various databases (data-mig, pre-prod, prod). From trainers, to testers, to business analysts, to developers and SMEs. Every person was being held up waiting for me to model and load data from a raft of different network types and topologies, some of which were still evolving as we were doing the migration. Data was the glue that held all the other disciplines together.

We were working with a tool that was very hierarchical in its data model. That meant that our data governance and migration plan was also quite hierarchical. But that suited the database (a relational DB from Oracle) and the network models (SDH, ATM, etc) available at that time, which were also quite hierarchical in nature.

When I mentioned “try, fail and refine” above, boy did I follow that sequence… a lot!! Like the time when I was modelling ATM switches that were capable of a VPI range of 0 to 255 and a VCI range of 0 to 65,535. I created a template that saw every physical port have 255 VPIs and each VPI have 65,535 VCIs. By the time I template-loaded this port-tree for each device in the network overnight, I’d jammed a gazillion unnecessary records into the ports table. Needless to say, any query on the ports table wasn’t overly performant after that data load. The table had to be truncated and re-built more sensibly!!

But I digress. This is a how-to, not a how-not-to. Here are a few hints to building a data governance strategy:

  1. Start with a WBS or mind-map to start fomalising what your data project needs to achieve and for whom. This WBS will also help form the basis of your data migration strategy
  2. Agile wasn’t in widespread use back when I first started (by that I mean that I wasn’t aware of it in 2000). However, the Agile project methodology is brilliantly suited to data migration projects. It’s also well suited to aligning with WBS in that both methods break down large, complex projects into a hierarchy of bite-sized chunks
  3. I take an MVD (Minimum Viable Data) approach wherever possible, not necessarily because it’s expensive to store data these days, but because the life-cycle management of the data can be. And yet the extra data points are just a distraction if they’re never being used
  4. Data Governance Frameworks should cover:
    1. Data Strategy (objectives, org structure / sponsors / owners / stewardship, knowledge transfer, metrics, standards / compliance, policies, etc)
    2. Regulatory Regime (eg privacy, sovereignty, security, etc) in the jurisdiction/s you’re operating in or even just internal expectation benchmarks
    3. Data Quality Improvement Mechanisms (ensuring consistency, synchronised, availability, accuracy, usability, security)
    4. Data Retention (may overlap with regulatory requirements as well as internal policies)
    5. Data Models (aka Master Data Management – particularly if consolidating and unifying data sources)
    6. Data Migration (where “migration” incorporates collection, creation, testing, ingestion, ingestion / reconciliation / discovery pipelines, etc)
    7. Test Data (to ensure suitable test data can underpin testing, especially if automated testing is being used, such as to support CI/CD)
    8. Data Operations (ongoing life-cycle management of the data)
    9. Data Infrastructure (eg storage, collection networks, access mechanisms)
  5. Seek to “discover” data from the network where possible, but note there will be some instances where the network is master (eg current alarm state), yet other instances where the network is updated from an external system (eg network design being created in design tools and network configs are then pushed into the network) 
  6. There tend to be vastly different data flows and therefore data strategies for the different workflow types (ie assurance, fulfilment / charging / billing, inventory / resources) so consider your desired process flows
  7. Break down migration / integration into chunks, such as by domain, service type, device types, etc to suit regular small iterations to the data rather than big-bang releases
  8. I’ve always found that you build up your data in much the same way as you build up your network:
    1. Planning Phase:
      1. You start by figuring out what services you’ll be offering, which gives an initial idea about your service model, and the customers you’ll be offering them to
      2. That helps to define the type of network, equipment and topologies that will carry those services
      3. That also helps guide you on the naming conventions you’ll need to create for all the physical, logical and virtual components that will make up your network. There are many different approaches to naming conventions, but I always tend to start with ITU as a naming convention guide (click here for a link to our naming convention tool)
      4. But these are all just initial concepts for now. The next step, just like for the network engineers, is to build a small Proof of Concept (ie a small sub-set of the network / services / customers) and start trialling possible data models and namings
      5. Migration Strategy (eg list of environments, data model definition, data sources / flows, create / convert / cleanse / migration, load sequences with particular attention to cutover windows, test / verification of data sets, risks, dependencies, etc)
    2. Implementation Phase
      1. Reference data (eg service types, equipment types, speeds, connector types, device templates, etc, etc)
      2. Countries / Sites / Buildings / Rooms / Racks
      3. Equipment (depending on the granularity of your data, this could be at system, device, card/port, or serial number level of detail). This could also include logical / virtual resources (eg VNFs, apps, logical ports, VRFs, etc)
      4. Containment (ie easements, ducts, trays, catenary wires, towers, poles, etc that “contain” physical connections like cables)
      5. Physical Connectivity (cables, joints, patch-leads, radio links, etc – ideally port-to-port connectivity, but depends on the granularity of the equipment data you have)
      6. Map / geo-location of physical infrastructure
      7. Logical Connectivity (eg trails, VPNs, IP address assignments, etc)
      8. Customer Data
      9. Service Data (and SLA data)
      10. Power Feeds (noting that I’ve seen power networks cause over 50% of failures in some networks)
      11. Telemetry (ie the networks that help collect network health data for use by OSS)
      12. Other data source collection such as security, environmentals, etc
      13. Supplementary Info (eg attachments such as photos, user-guides, knowledge-bases, etc, hyperlinks to/from other sources, etc)
      14. Build Integrations / Configurations / Enrichments in OSS/BSS tools and or ingestion pipelines
      15. Implement and refine data aging / archiving automations (in-line with retention policies mentioned above)
      16. Establish data ownership rules (eg user/group policies)
      17. Implement and refine data privacy / masking automations (in-line with privacy policies mentioned above)
    3. Operations Phase
      1. Ongoing ingestion / discovery (of assurance, fulfilment, inventory / resource data sets)
      2. Continual improvement processes to avoid a data quality death spiral, especially for objects that don’t have a programmatic interface (eg passive assets like cables, pits, poles, etc). See big loop, little loop and synchronicity approaches. There are also many other data quality posts on our blog.
      3. Build and refine your rules engines (see post, “Step-by-step guide to build a systematic root-cause analysis (RCA) pipeline“)
      4. Build and refine your decision / insights engines and associated rules (eg dashboards / scorecards, analytics tools, notifications, business intelligence reports, scheduled and ad-hoc reporting for indicators such as churn prediction, revenue leakage, customer experience, operational efficiencies, etc)
      5. In addition to using reconciliation techniques to continually improve data quality, also continually verify compliance for regulatory regimes such as GDPR
      6. Ongoing refinement of change management practices

I look forward to seeing the TM Forum report when it’s released later this month, but I also look forward to hearing suggested adds / moves / changes to this OSS / BSS data governance article from you in the meantime.

If you need any help generating your own data governance framework, please contact us.