This document briefly discusses the process of implementing a Cubeia CEP installation for monitoring and extracting real-time business intelligence from an existing customer system. It does not cover explicit data monitoring tools such as dash boards or data mining.

All examples in this article are simplified the sake of brevity.

1 - Establish Project Goal

It is important that a project implementing a Cubeia CEP installation has clear goals. This step attempts to answer the basic question: What do we want to achieve?

Cubeia believes in Agile methodologies, and recommends that the first focus for the Cubeia CEP project is a single goal, which can be easily expressed in one sentence. This goal oriented approach unifies the team, and provides a common target, enhances team focus, and provides a do-ability check for the Customer. Example goal sentences may be:

• Monitor and correlate system CPU and bandwidth load to warn the support teams when the average load exceeds a configurable threshold for a given amount of time.
• Identify when a client have had a “bad streak” by performing less than his average in several poker hands in a row, and flag this client for a back office operator.

2 - Identify Data Sources

We need to analyze the Customer system, and identify sources of data and what form that data has. This could be log files, Java Management Beans, message queues or purely internal points in the code. Crucially, this step establishes:

• From where will the data be sourced?
• How will the data be extracted from the source?
• What data will be pulled from the source?

There may be several sources,and the format of the data may differ. Using the examples from step one, we may establish the following sources:

• A script polling a system server every five seconds for CPU levels (user, system, idle and wait), and the currently handled bytes per second in and out of one of the networks interfaces.
• A customized code within the poker server, reporting the hand result for each player when the hand is finished.

3 - Identify Data End Points

What data are we looking at extracting, and what format does it have? This step attempts to answer what the system will deliver and identifies the “trigger” which satisfies the goal established in step one in terms of data delivery. For example:

• Send an SMS to a configurable mobile number when the values exceeds a configured threshold. The SMS should contain a time indication, and a summary of the current values.
• When a client is identified as having a “bad streak”, put the client ID on an MQ (to be read by the back office modules).

4 - Define CEP Algorithm

In this step, the question “how can we, and when computation steps do we need, to transform the sourced data to trigger the end data” is answered. This is a semi-technical step where Cubeia personnel helps the Customer identify the algorithm, and stages needed to transform and handle the input data.

For a system monitoring CPU and bandwidth over time, the input data would be polled in intervals, in other words arriving as “events” from different server in the system. The CEP algorithm would most likely be one or more CEP queries over a sliding time window, correlating the data and keeping an average over all servers.

Identifying a “bad streak” for a given poker client would specify what circumstances constitutes such a warning, and then specify one ore more sliding time window query taking the sourced data and processing them for every and each client.

Crucial for every CEP algorithm is the ability to correlate data over time. Cubeia CEP uses a library called Esper for this, and Cubeia will assist the Customer in identifying the processing needed for this step.

5 - Define Data Carrier Objects

Now it starts to get technical. Together with the Customer technical staff Cubeia will produce specifications for the data identified in step two and three. This specification will must likely end up in a neutral format such as an XML schema or a simple text document. For example:

• An XML schema defining a simple XML format for system CPU and bandwidth status. Plus a text document detailing the contents of a warning SMS for the system administrators.
• An UML diagram for data objects carrying a poker hand result, and a client ID object.

This step is more or less intertwined with the next step, as production of carries objects is closely related to transformation and transport.

6 - Define Arena Entry and Exit Points

The Cubeia CEP installation will be installed on separate computers for isolation and scalability. This step identifies how the data carrier objects enters and exists the arena. Also, there may be transformations involved as the CEP stages will have to use normalized data to operate efficiently.

The most common entry point into a Cubeia CEP installation is by utilizing message queues. This is an industry standard and has good support in all major programming languages. But Cubeia CEP also supports XML over HTTP and many other standards.

The exit point depends entirely on the goal of the project, and may have to be completely customized by Cubeia for the Customer. But common exits are message queues, databases and remote calls using web services.

7 - Implementation Phase

This is when it is all put together. Personnel from Cubeia and the Customer will work together to assemble the CEP installation given the knowledge gathered from the previous steps. This includes implementing the data sources, implementing the processing stages, installing and configuring the Cubeia CEP servers, optionally installing and configuring connecting servers such as message queues, and testing the initial installation.

This step will be where the majority of the time spent will end up. It's length is determined by the complexity of the setup and the number of resources invested. Also, it is anticipated that for complex operations this phase may be performed by several teams in parallel.

8 - Optional Considerations

So far we've concentrated on the data analysis process for a Cubeia CEP installation. However, there are more components that may need to be produced. For example:

• Does the information need to be presented in real time? Cubeia has in its tool box components for building a dash board over a transient flow of data and has experience of building monitoring applications for industry leading companies.
• Should the data be stored? For many applications the events used for a CEP arena are too numerous to be stored in a normal relational database. Cubeia can help the Customer apply cutting edge storage solutions such as distributed storage, column databases and map/reduce style data mining to harness the power of the collected information.
• When the first, limited goal is achieved, what is the next goal? Cubeia can help the Customer identify business critical data hidden in the system, but which could be harnessed for powerful BI-processing using the already in place CEP arena.

Parting Words

Hopefully this article has given you and idea on the steps included when going with Cubeia CEP for your system. It should be pointed out though, the implementing CEP is individual for each and every Customer. When it comes to this kind of business intelligence we don't believe in over the counter solutions, we believe you deserve better than that.

Lars J. Nilsson is a founder and Executive Vice President of Cubeia Ltd. He's an autodidact programmer, former opera singer, and lunch time poet. You can contact him at lars.j.nilsson in Cubeia's domain.