Statistical Analysis of Massive Data Streams: Proceedings of a Workshop (2004)
Chapter: 7. STREAM RULE PROCESSING
Figure 4. The events of the stream are converted into Normalized Metered Events and passed directly to a rule engine.
Once collected, the data of an ME are normalized into a common data structure called a Normalized Metered Event (NME, see Figure 4), which is an array of attributes that contain different data types similar to a DB record. The only attribute that is required is an end-time stamp. All the other attributes can be configured to suit the processing needs of the application. Unlike DB record schemas the number and type of attributes can change dynamically as it undergoes processing. In this stream processing context new attributes can be dynamically “adorned” to the NME and be used as intermediate variables, which travel along with the NME in the stream, and then disposed of when no longer needed. (The word “travel” is only a metaphor. The NME object doesn’t actually move; its position in the stream is tracked by passing a small reference or “pointer” to the NME object from rule to rule.)
7. STREAM RULE PROCESSING
Once normalized, the NMEs move directly into a Rule Engine, which has been specifically designed for merging, processing, and splitting streams.
The input streams can be independent or related (such as a usage stream and a session stream). In the rule engine (Figure 5) there can be multiple rule chains that operate on the input streams. It is possible to have a single stream processed by multiple rule chains each producing distinctly different output streams, or multiple similar streams processed by a single rule chain, or combinations of the above.
There are several forms that the output streams can take: (1) During processing of an NME, attributes within the NME are added, modified, or deleted and the result NME is forwarded immediately to a downstream Collector for further processing. (2) A cyclic aggregation time interval is configured into the Collector and rule chains are configured to
