Visualization and Analysis of Massive Internet Traffic Data

Data collection on high speed Internet links can result in large databases. To exploit very large databases we need to do more than computing simple summary statistics; we need to study the data in detail and in its full complexity. To help achieve this we developed S-Net, a traffic measurement and analysis system that begins with packet header collection on network links, and ends with data analysis on a cluster of linux PCs running S, Splus, or R. The system provides tools for the study of packet header data with detailed comprehensive characterization, visualization, statistical modeling and simulation. When used in real-time, the system enables network performance monitoring and anomaly detection for network traffic. Currently, the system has been used at two network locations, one at a Bell Labs Research network and the other at the NCNI Research facility as part of the DARPA Helios project.