Real-time sandpits

You and I are streaming data engines.”
Jeff Hawkins

You’ll have noticed that I often write about using sandpit environments to trial and refine different OSS configurations. They tend to be really helpful as a change management facilitation mechanism. One of the things you might be wondering is exactly what can you do in a sandpit? How does the data get into it to play with?

Some vendors will be able to provide you with a minimally populated OSS straight out of the box, that may have some reference data pre-configured. Examples might include types of physical ports (eg RJ45, etc), bandwidths (eg 1Mbps, etc) and many more. Others might be completely empty. Others might not be able to provide even a basic workable configuration without heavy modification by their services teams. Either way, it’s important for your project team to get involved and understand what it takes to build up a partially configured system.

You also need to determine where you’re going to get data from so that you can try various configurations. Data could come from various sources, including:

  • Manually created
  • Bulk / script loaded
  • Simulated streamed data
  • Data from real data sources (static or streamed)

I usually start with the manually created data by establishing a set of scenarios that I’d like to model in the sandpit. This could be a basic network (I often start with a transmission network like an SDH ring) and then build other network / service layers on top (eg IP network, VPN service). This helps to model inventory, naming conventions, service types, order entry, knowledge-bases, etc.

This partially loaded data then allows me (or the vendor) to interrogate their data model and determine how to bulk load data using scripts. This sometimes requires reverse engineering of database schema… Eeek! This may allow you to quickly load larger networks from data you already have in CSV files or spreadsheets though.

Manual creation of data is often not suitable for real-time tools like alarm/fault or performance managers. For the purpose of product demonstration, the vendors will often have simulators that populate pseudo-real-time data into their tools. This could take the form of randomly generated events, or script-loading data from CSV files at certain time intervals, or various other approaches. Irrespective, it will allow you to trial various configurations of reports, workflows, naming conventions, etc. Often the simulators will allow for customisation of data to represent different network configurations and the vendors may allow you to tweak the simulated data for your specific needs. The advantage of simulators is to generate pseudo-real-time traffic without having to integrate live network feeds, which can be time-consuming to set up.

If you want to take your sandpit to the next level, you can then consider integration of data from real data sources via APIs and real devices. If you have a lab environment of your network devices, this could be quite feasible to integrate these with your sandpit. I’ve even seen some cases where production data feeds have been multi-homed to sandpit environments. Note that this stage is usually a secondary consideration because of the effort required to integrate real devices and you’re now beginning to test the integration rather than the functionality of the OSS tools.

IMPORTANT NOTE: It’s possible that you’ll have to specify some of these aspects up-front in contract discussions because some vendors may baulk at assisting you to establish an early release sandpit after contracts have been agreed.

Read the Passionate About OSS Blog for more or Subscribe to the Passionate About OSS Blog by Email

Leave a Reply

Your email address will not be published. Required fields are marked *