|
Complex Event Processing: Leveraging Intelligence From Massive Amounts of Data
By Richard Tibbetts
The need to leverage intelligence from the massive amount of data hitting organizations today is no longer a nice-to-have, but is rapidly becoming essential to maintaining competitive advantage. It’s time to get serious about the data deluge, and to transform this data into intelligence.
Traditional relational databases and data warehouses provide business intelligence (BI) that is of great value for conventional analytic applications - where business decisions need to be made today, this hour, or within minutes. However, these traditional database resources cannot effectively support a new and growing class of fast-paced business applications where decisions must be made this second, or within milliseconds.
Instantaneous reaction to real-world event data can make a significant difference in a variety of applications such as stock market trading. More data is being generated faster than ever before in stock exchanges, in part because of global market and regulatory changes. Traders that can run quantitative analytics and automatically trade from high-volume, high-speed data feeds gain critical competitive advantage.
The need to react and respond instantly to high-volume, high-speed data for competitive advantage, to minimize risks, and avert disasters has never been greater. Thus, a revolutionary technology is needed to perform complex analytics on high-speed, high-volume data to enable instantaneous response. That technology is high-performance complex event processing (CEP).
High-performance Complex Event Processing
High-performance CEP is the continuous processing, analysis, and response to high-volume, high-speed data streams in real-time. It can be used to identify and analyze cause-and-effect relationships among events. And because it does all of this in real-time, it enables instant and informed responses to critical intelligence in real-time. Four different approaches are typically considered for high-performance CEP:
- Custom-Coded Applications - typically written in Java, C++ or other similar programming languages. Custom applications had been the leading solution for many organizations until the recent advent of off-the-shelf CEP. However, custom coding is limited by long development cycles of 6-9 months or more, the high cost of maintenance, and inflexibility.
- Rules Engines - for watching input data-streams for conditions of interest that match its rules - typically condition/action pairs, usually expressed using “if-then” notation. The rules are typically written in a proprietary language and when a condition of a rule is matched, the rule is said to “fire” a corresponding action, alert or output to external applications.
- In-Memory Databases - to reduce the latency delays of traditional database management systems by eliminating disk I/O, caching, and transaction logs. But they still incur latency penalties and require custom-coding to build CEP applications.
- Stream Processing Software - handles streaming data and performs queries and computations on incoming messages as they fly by - without storing them. Stream processing uses specialized primitives and constructs, such as time-windows, to express stream-oriented processing logic.
Of all of the commercial offerings, high-performance CEP using stream processing software offers the lowest latency, highest throughput, most flexible programming model, and fastest time-to-value.
High-Performance CEP Platforms vs. Point Solutions
Two types of high-performance CEP software are generally available: platforms and point solutions. In the same way that a commercial relational database includes core platform capabilities, which allow building a variety of server-based applications, high-performance CEP platforms have a number of fundamental characteristics that enable developers to build applications across a range of industries and problems.
To process information in real-time with the lowest latency, a CEP platform must perform message processing without first storing and retrieving the data. The highest performance CEP platforms do this by using a language such as StreamSQL, which extends standard SQL via time or event-based windows to readily execute queries and perform pattern-matching functions on streaming data. CEP platforms also must store and access current or historical state information, preferably using a familiar standard such as SQL.
In addition to these characteristics, a CEP platform should have the following capabilities to provide the necessary framework for building a variety of applications within and across multiple industries:
- Predictable Outcomes: In a real-time processing infrastructure, the system must process time-series records in a consistent, deterministic manner to ensure that running the same dataset through the system multiple times always yields the same results.
- Data Safety and Availability: If a failure occurs in a CEP application - regardless of the cause (hardware, operating system, software, application) - the application needs to continue operating without losing any data.
- Automatic Scaling and Partitioning: As the volume of real-time information continues to expand, applications for processing this data must be able to scale quickly and automatically.
- Flexible, Graphical Programming: Since much of the underlying CEP technology is complicated in nature, stream processing development should be available to business users and IT professionals at a high level of abstraction via a drag-and-drop graphical workflow development paradigm.
- Security: As CEP becomes more broadly integrated throughout the enterprise, users can reasonably expect these systems will provide appropriate security capabilities and integrate with existing systems.
- Interoperability with Enterprise Systems and Data: The system must be able to process a range of data types and formats including structured and unstructured data, financial market data, IP logs, email, XML data and files, RSS feeds, and comma-separated value (CSV) files.
Alternatively, and on a smaller scale, CEP point solutions provide complex event processing for specific limited application areas, such as only one form of algorithmic trading. In general, these solutions have a proprietary programming language and fixed or pre-defined data models, user interfaces, and analytics.
Typically, businesses acquire CEP point solutions to solve a specific problem limited to a departmental silo. However, businesses that see the potential for CEP to benefit multiple departments or applications within their organizations will look to a high-performance CEP platform that offers flexibility, broader adaptability to various business challenges, and cross-enterprise interoperability. Many applications can be supported by one high-performance CEP platform - typically at lower cost per application than if individual point solutions were employed.
Real-World CEP: Transforming BI Applications
High-performance CEP is transforming how organizations in financial services and other industries acquire, process and manage real-time as well as historical event data. For example, financial services firms leverage high-performance CEP to monitor data coming into their ticker plants and feed handlers. They also use CEP to trigger alerts if latency drops below acceptable levels, normalize symbology, filter massive data streams to pass only selected data downstream, route the first arriving quote from across multiple exchanges to traders, and calculate real-time statistics on quality of service from feeds. Additional financial services applications include algorithmic trading, regulatory compliance, real-time P&L, and enterprise risk management.
Other industries also face the need to process and understand large quantities of data over short periods of time, and with the competitive landscape growing ever-more challenging, the time to act is right now. High-performance CEP is the solution - and it’s the only technology that can deliver on that promise.
About the Author
Richard Tibbetts, chief architect, StreamBase Systems, is responsible for the design and implementation of the StreamSQL programming language and the high-performance, low-latency server components of the StreamBase Event Processing Platform. www.streambase.com
|
