Statistical Analysis of Massive Data Streams: Proceedings of a Workshop (2004)
Chapter: 1. INTRODUCTION
A Stream Processor for Extracting Usage Intelligence From High-Momentum Internet Data
Lee RHODES
The data streams of the Internet are quite large and present significant challenges to those wishing to analyze these streams on a continuous basis. Opportunities for analysis for a Network Service Provider include understanding subscriber usage patterns for developing new services, network demand flows for network operations and capacity planning functions, and early detection of network security breaches. The conventional analysis paradigm of store first, then analyze later has significant cost and latency issues when analyzing these high-momentum streams. This article presents a deployed architecture for a general purpose stream processor that includes dynamically configurable Capture Models that can be tailored for compact collection of statistics of the stream in real time. The highly configurable flow processing model is presented with numerous examples of how multiple streams can be merged and split based on the requirements at hand.
Key Words: DNA; IUM; Real-time statistics; Statistical pre-processing.
1. INTRODUCTION
In 1997 a small R&D group was formed inside of Hewlett-Packard’s Telecommunications Business Unit to develop Internet usage management software for Network Service Providers (NSPs). The services offered by these NSPs ranged from Internet backbone to Internet access. The range of access services included residential and commercial broadband (cable and xDSL), dial-up, mobile data, as well as numerous flavors of hosting and application services. Early on our focus was the processing of usage data records (e.g., NetFlow® or sFlow®) produced by Internet routers. However, it quickly broadened to include convergent voice Call Detail Records (CDRs) as well as the ability to collect and process data from a very broad range of sources such as log files, databases, and other protocols. The diverse technological histories (and biases) of the different segments of the communications industry created for us interesting challenges in creating a software architecture that was
Lee Rhodes is Chief Scientist/Architect, IUM/DNA, Hewlett-Packard Co. (E-mail: lee.rhodes@hp.com).
©2003 American Statistical Association, Institute of Mathematical Statistics, and Interface Foundation of North America
Journal of Computational and Graphical Statistics, Volume 12, Number 4, Pages 927–944 DOI: 10.1198/1061860032706
