Distributed packet capture probes
Because every packet of target flow classes has to be individually monitored, trafMon uses distributed packet capture probes. This allows to collect timestamp observations, to dissect the stack of protocol headers and to analyse the protocol conversations of the actual traffic flows without modifying their behaviour.
A same probe can have multiple packet capture interfaces so as to gather observations at different local stages of the network path and/or at different branches of alternative routes. Of course, this also permits to cover different data flows when they do not appear at a single probing point in the infrastructure at a site.
Single or bi-directional measurements
For those bi-directional data flows measurements, a single probe suffice to collect traffic volumes and other protocol counters, to follow the conversations exchanges and to measure the round-trip time of a network service.
Uni-directional latencies of the traffic in each direction are also of interest. This implies then to place a probe at least at the peer ends, and maybe at intermediate hops over the flow path.
Custom determination of measurements applicability
A single XML configuration file defines, for all trafMon online components (probes and collector(s)), which flow patterns are to be applied which measurement analyses. It also defines the rules to partition the monitored traffic into different flow instances. It permits to elect a single packet as belonging to different flow instances (e.g. uni and bi directional variants) members of different flow measurement classes.
Data centralisation protocol
A home-made protocol, on top of UDP, let all the probes forward every type of their observations to one central collector component (or even multiple ones in a resilient deployment). This has been designed to resist to quite severe transmission conditions. It can be rate controlled, with maximum size limit applicable to the protocol data units, in order to permit its injection together with real time application traffic under stringent performance requirement (e.g. safety-of-life remote surgery communications).
Of course, it is added reliability with configurable timeout and retry parameters, as well as checksum verification and quite small acknowledges in reverse direction. This protocol allows both the probe and the collector to detect the break of access to a peer. In such condition, the probe will preserve those types of information that are considered important (definitions of newly discovered flows, records of FTP transferred files, monitored protocol troubleshooting details just before the detected break of connectivity), for perpetual retrying at very slow pace, until an ACK is received from the collector.
Designed for high traffic volumes analysis in real-time
The probe is split in two processes interconnected by a simple circular buffer. Both processes are running in true parallelism on different CPU cores. The probe father process performs zero-copy protocols dissection of the kernel resident captured packets content. The sieve that determines which analysis must be conducted on each packet, and to early skip all those not relevant packets is particularly optimized to traverse the decision tree in one single pass.
Only those necessary elements of protocol information and, sometimes, of packet content, are transmitted to the probe child process. Several optimised data structures permits to keep track of the flow state information and of the measurements observations.
Per-packet one-way monitoring thanks to unique signature and possibly multiple timestamps
For some recurring data flows (e.g. SNMP monitoring, NTP polling, DNS queries/replies …), the trafMon administrator can specify the precise monitoring of uni-directional packets or re-assembled datagram units. Each probe then hash the packet (or datagram) content (its part that would change over the network travel, for coping with NAT/PAT network/port address translation) and keeps a few bytes (1 to 5) as identifying signature.
The probe can collect several timestamps per observed data unit:
- the individual packet capture time, or those of the first and last seen IP fragments,
- optionally, the timing information from specially crafted IP packets with the TIMESTAMP option present: stamps of the gateways over the network route,
- and/or, for NTP requests or responses, the user-specified timestamps embedded in the application layer NTP payload.
Records of per packet/datagram (flowID, signature, size and timestamp(s)) are piled by the probe using a quite compact encoding, so as to limit this observations volume to less than 1% of the monitored traffic. The collector gradually reconciles the partial observations delivered by the (several) probes on the flow path. A specially implemented data structure, combining a balanced search tree (2-3 BTree) capable of “near sibling” retrieval – for matching a “quite-simultaneous” stored record – and a doubly-linked list – for easy traversal – permits the collector to find the per-packet/datagram record with observations already received from other probe(s) and complete it. Thanks to this sophisticated algorithm, collisions of signature hash can be detected and coped with at best.
Experience shows that 2 bytes of signature leads to too frequent clashes. 3 bytes is a good compromise provided one can cope with about one clash per day, while 4 bytes seem to suffice for avoiding any clash in relatively sustained traffic over a week. This parameter influences the relative size of monitoring data possibly injected online through the same network link as the time-critical observed data flows. The shorter this information, the least is the risk of disturbance by the monitoring activity.
After a configured timeout, although coping with potential late delivery from a “disconnected” probe, those still incomplete observations indicate the packets lost over their travel. the trafMon administrator has the choice of producing simple per packet records with the sequence of observed (or missing) timestamps, or he can let the collector measure the latency (aggregated as an histogram distribution per reported period) between to hops, and to separately produced the counters 0f lost or missed packets/datagrams (or those whose observations have been dropped by a silent probe).
Preserving detailed protocol monitoring observations
The probe extracts every possible item of information out of the lower layers protocol headers (IPv4, TICMP, UDP, TCP), to accumulate a long list of counters that are aggregated and reported at user-defined frequency for user-specified flow classes.
TCP connections can be selectively measured, including all details about the detected retransmissions as well as the maximum and last window sizes.
All command/response exchanges over FTP control sessions are reconstructed. And data connections (for directory listing of GET/PUT file transfers) in active and passive mode are matched. Individual file transfers are described and FTP counters complete those of the lower layers.
Being for TCP (SYN/ACK or based on RTTM option), for ICMP Echo or for UDP-based request/response transactions (SNMP, NTP, DNS), the round-trip time from the probe and back are measured and an histogram distribution is built for every reporting period. For NTP, the round-trip delay with initiator (NTP client) reflects the NTP poling period; an increase in frequency indicates a degraded time synchronisation that should be investigated.
Collector output as simple CSV text files
All observations produced by the probes, as well as those uni-directional measurements (latency, packet losses) computed by the collector are simply written to timestamped ASCII log files as a series of lines with ‘|’ separated fields (CSV). Some users could simply exploit these data in a custom way with their own tools.
trafMon however provides a Python script and a collection of database stored procedures to efficiently load them in a MySQL database for updating aggregates at 1 minute, 1 hour and 1 day granularity.
This script preserves the loaded raw data into an archived of compressed tarfiles (.tbz) that the trafMon administrator can can age-out as needed.
For those using the optional CERT™ SiLK additional data feed from NetFlow/sFlow/IPFIX, another Python script is provided that extracts, every hour, those flow volumes information records, as another type of CSV text file. These records are sliced into per-minute values and loaded together with the trafMon collector logs.
Also at this point, the user is able to custom exploit the time-aggregated tables of protocol counters, the distribution histograms of IP sizes and of one-way and two-way delays, the per TCP connection records and those describing the individual FTP file transfers.
Custom-defined qualification of peers IP addresses
For those known IP host addresses and LAN address segments, belonging to the Organisation where trafMon is deployed, the tool administrator can specify a partitioning in terms of relevant Organisation activities, and another in terms of locations. All IP addresses can also undergo a reverse DNS lookup, retrieving their fully-qualified domain name (FQDN). For other peer addresses, the Maxmind® GeoIP™ pre-loaded registry is used for assigning an ASN number, a country and a city.
Once a day at night, the flow volume information (from trafMon IP counters and, optionally, from the NetFlow based information) can be prepared, with flow qualification in terms of activities/locations, country, FQDN and service protocol name.
Eclipse BIRT flexible database reporting and pre-defined set of report templates
As a complete pre-built product, trafMon comes with a set of graphical report templates with embedded text, tabular presentations and business charts. These report layouts have been draw with the Eclipse BIRT Designer editor, and can be adapted or complemented by the user.
One set of pre-designed templates present the several protocol details raw measurements, in support of communications diagnostics and troubleshooting activities.
Another set of reports show the way the Organisation network services are actually used overs activities and/or locations and what are the peer (activities/locations or countries), and what are the user perceived performances (overall KPI’s for FTP and TCP)