Software Components and Usage
Executables are located in the /opt/trafMon/bin and /opt/trafMon/testing directories:
Some MySQL stored procedures, provided in the trafMon_template database, are also for direct use:
The most important commands are explained here below.
1. The Probes
The probe executable has the following usage:
tmon_probe [-l] [-c configXML] [-n NEWconfigXML] probeName
USAGE: -l (local) means using ./tmon.xml, ./tmon-new.xml and, if it exists, ./tmon_probe.diag -c means use the given XML, and nothing else -n monitors the given NEW XML for scheduled config update (based on its `startAt' attribute If -l is NOT given, /etc/trafMon/xml/tmon.xml is used If -l and -n are NOT given, /etc/trafMon/xml/tmon-new.xml is looked at If -l is NOT given, or ./tmon_probe.diag doesn't exist, /etc/trafMon/diag/<probeName>.diag is used
2. The Central Processing System
2.1 The Collector
The collector executable has the following usage:
tmon_collector [-l] [-c configXML] [-n NEWconfigXML] collectorName
USAGE: -l (local) means using ./tmon.xml, ./tmon-new.xml and, if it exists, ./tmon_collector.diag -c means use the given XML, and nothing else -n monitors the given NEW XML for scheduled config update (based on its `startAt' attribute If -l is NOT given, /etc/trafMon/xml/tmon.xml is used If -l and -n are NOT given, /etc/trafMon/xml/tmon-new.xml is looked at If -l is NOT given, or ./tmon_collector.diag doesn't exist, /etc/trafMon/diag/<collectorname>.diag is used
2.2. The scripts
This section describes the different scripts used by the trafMon software for a variety of tasks. The scripts are located in /opt/trafMon/bin/ unless mentioned otherwise explicitly.
Options: -h, --help show this help message and exit -p PATH, --path=PATH Path to the directory containing the data file to load into the DB (default: /var/trafMon/collector/ -l, --localConfig If -l is specified, the db.cred file in the current directory is used. Otherwise, the default /etc/tmon/cred/db.cred file is loaded -L LOGFILEDIRECTORY, --logFileDirectory=LOGFILEDIRECTORY Path to log directory (default: /var/trafMon/log/)
This script loads the probe and collector data, and optionally the SiLK NetFlow records, into the MySQL database.
Bulk load in temporary tables, then update persistent lists and persistent counters aggregates at 1 minute, 1 hour and 1 day.
Options: -h, --help show this help message and exit -p INFOFILE, --path=INFOFILE File pathname to the .ini file with information about the known IP adresses/segments. (Default: /etc/trafMon/ipInfo.ini) -l, --localConfig If -l is specified, the db.cred file in the current directory is used. Otherwise, the default /etc/tmon/cred/db.cred DB configuration file is loaded -g GEOIPPATH, --geoipPath=GEOIPPATH Path to the directory with GeoLite2 or Maxmind GeoIP2 databases (Default: /var/trafMon/GeoIP/) -a, --all Activate a full update of the ipInfoTable rather than a partial one (partial means that we do not try to resolve addresses which have already been successfully resolved previously). -L LOGDIR, --logFileDirectory=LOGDIR Path to log directory (default: /var/trafMon/log/)
This script fills the MySQL “ipInfotable” table relating an IP with a location, an activity and/or a country, a city and an ASN, and performs a DNS reverse lookup.
It applies to unsolved IP addresses (typically those freshly discovered; or, with –all, it conducts a complete re-scan of all registered IP addresses.
MAXMIND® free GeoLite2™ IP geolocation database files
The MaxMind Company sells GeOIP registries. It is also providing free access to GeoLite2 semi-accurate version of their commercial product. But, since the legal enforcement of privacy protection of personal data (in particular in California and the EU), users must first register at their site before getting the database files: https://dev.maxmind.com/geoip/geoip2/geolite2/. Please carefully analyse the licensing conditions. The databases used to obtain information about an IP should be downloaded or copied in /var/trafMon/GeoIP/.
Previously, the following commands could be used while being in the said directory:
– wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
– wget http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
– gunzip *.gz
These commands downloaded the two database files used by the script and decompress them.
Options: -h, --help show this help message and exit -H nbOfHours, --hour=nbOfHours Number of hours of data to process (starting from the past hour). This option has precedence to the start/end date options. E.g.: '-H 2' at 14:36:27 means data for [12:00:00, 14:00:00] -S startTime, --dateStart=startTime Time at which the collection of data will begin (format: YYYY/MM/DD:HH). The data for the given hour are included in the results. -E endTime, --dateEnd=endTime Time at which the collection of data will end (format: YYYY/MM/DD:HH). The data for the given hour are included in the results. -N SRCPATH, --netflowdata=SRCPATH Path to the root of the SiLK collected NetFlow data (default: /var/silk/data/) -D DSTPATH, --collectordata=DSTPATH Path to the runtime observations logs produced by the trafMon Collector, where to place the NetFlow extracted log (default: /var/trafMon/collector/) -C SILKCONFIG, --silkconfig=SILKCONFIG Path to SiLK config file (default: /var/silk/twoway- silk.conf) -s SILKBIN, --silkbin=SILKBIN Path where to find SiLK rwfilter and rwcut utilities (default: /usr/local/bin/) -L LOGDIR, --logFileDirectory=LOGDIR Path to log directory (default: /var/log/trafMon/)
This script retrieves, every hour, the NetFlow records received and stored in the SiLK tree of raw log binary log files. The resulting text file is added to the trafMon collector output, subject to next run of trafMon_loader.py.
Options: -h, --help show this help message and exit -d DBname, --db=DBname Database to be used. Default to 'trafMon'. -r report, --report=report Type of synthesis report to be generated. Possible choices are: [manager, operator, conversation]. Default to 'manager'. -D destination, --destination=destination Destination directory. Default to '/var/trafMon/reports/2020/11/13', where the 'YYYY/MM/DD' part is the generation time of the report (today). -t top, --top=top Top-N to be used. Possible choices are: [5, 10, 15, 20, 25]. Default to top-5. -T threshold, --threshold=threshold Threshold bandwidth in b/s to be used. Possible choices are: [0, 1000, 10000, 50000, 100000, 500000]. Default to 1000. -A activityName, --activity=activityName Activity to be used. Default to 'any'. Use quotes if the activity name include a space. -L locationName, --location=locationName Location to be used. Default to 'any'. Use quotes if the location name include a space. -H hostName, --host=hostName Host to be used. Default to 'any'. Use quotes if the host name include a space. -s startDate, --startDate=startDate Start date to be used (format: 'YYYY-MM-DD'). Default to first day of previous month. -e endDate, --endDate=endDate End date to be used (format: 'YYYY-MM-DD'). Default to last day of previous month. -l LOGDIR, --logFileDirectory=LOGDIR Path to log directory (default: /var/log/trafMon/) -R TEMPLATESFOLDER, --reportTemplatesDirectory=TEMPLATESFOLDER Path to trafMon report templates directory. Default: /opt/trafMon/trafMon_reports/ -g GENREPORT, --genReport_sh=GENREPORT Full pathname to the Birt runtime 'genReport.sh' utility. Default to /opt/trafMon/bin/genReport.sh, which is typically a symbolic link to the BIRT RunTime installation/ReportEngine/genReport.sh
This script is used to automatically produce a sysnthesis PDF report.
Options: -h, --help show this help message and exit -l, --localConfig if -l is specified, db.cred file is fetched from the current directory. Default: /etc/trafMon/cred/ -f FILENAME, --filename=FILENAME Give a pathname or filename containing IP address patterns in concerned Flow Instances. This file basename is also the root of the tree of generated reports. When relative, the file is fetched from current directory when -l is specified, otherwise from default /etc/trafMon/report/ -D destination, --destination=destination Destination directory. Default to '/var/trafMon/reports/'. -s STARTDATE, --startDate=STARTDATE Give a start date in format: 'YYYY-MM-DD' -e ENDDATE, --endDate=ENDDATE Optionally give an end date in format: 'YYYY-MM-DD' -t TIMESPAN, --timespan=TIMESPAN Without endDate: choose between 'weekly' or 'monthly' report. With endDate: give any identifier for this type of reports. -L LOGDIR, --logFileDirectory=LOGDIR Path to log directory. Default: /var/log/trafMon/ -T TEMPLATESFOLDER, --reportTemplatesDirectory=TEMPLATESFOLDER Path to trafMon report templates directory. Default: /opt/trafMon/report/ -g GENREPORT, --genReport_sh=GENREPORT Full pathname to the Birt runtime 'genReport.sh' utility. Default to /opt/trafMon/bin/genReport.sh, which is typically a symbolic link to the BIRT RunTime installation/ReportEngine/genReport.sh
This script is used to automatically produce the set of protocol details PDF reports for IP hosts whose address match the configured patterns.
trafMon_probeResurrect.sh.sample “<tmon_probe start cmd>” <logfile_pathname>
trafMon_serverResurrect.sh.sample <log_pathname> [(“” | “<tmon_coll start cmd>”)
[(“” | “<msqld start cmd>”) [“<tomcat start cmd>”]]]
These sample scripts can be adapted and scheduled every minute by the crontab or the appropriate Linux accounts, on the appropriate computers, to automatically launch (or re-launch after potential crash) the key daemon processed of an operational trafMon installation.
3. MySQL Stored Procedures
Update_ftp_data_in_ipcttable( _dbName, _rangeStart, _rangeEnd)
Assigns the volumes of actual FTP data connections to their original data flows, encompassing corresponding FTP control session.
Aggr_activityvolumetable_first_level( _dbName, _rangeStart, _rangeEnd)
For the given range of days (up to yesterday), or for yesterday when both rangeStart and rangeEnd are NULL, update the prepared tables, at 1 hour and at 1 day granularity, for the “quick” generation of synthesis reports on trafMon probe data: with every flow having full description of its peer IP systems, full volumes in each direction, and the corresponding FTP and TCP key performance values.
Aggr_activityvolumetable_netflow_first_level( _dbName, _rangeStart, _rangeEnd)
For the given range of days (up to yesterday), or for yesterday when both rangeStart and rangeEnd are NULL, update the prepared tables, at 1 hour and at 1 day granularity, for the “quick” generation of synthesis reports on SiLK gathered NetFlow data: with every flow having full description of its peer IP systems and full volumes in each direction.
Drops, from all existing trafMon database instances, the remaining working tables, produced upon the generation of BIRT report instances, and that cannot be declared as temporary tables (automatically drop at end of database connection).Those working table names start with an ‘_’.
Partition_drop( SCHEMANAME, TABLENAME, NB_DAYS)
Cleanup the given trafMon database from obsolete fine-gain observations from the given table name (stored in separate physical partitions of a same logical table), while preserving the last NB_DAYS days of data.
4. Traffic Generators
trafgen [-l] [-c XML] tgenName
USAGE: -l (local) uses ./.xml and if it exists, ./trafgen.diag -c means use the given XML, and nothing else If -l is NOT given, /etc/tmon/xml/<tgenName>.xml is used If -l is NOT given, or ./trafgen.diag doesn't exist, /etc/tmon/diag/<tgenName>.diag is used
Conducts fully controlled repetitive scenarios, defined in an XML configuration file, of sending specially crafted UDP and/or TCP packets.
ftpgen.py host user password number-of-simultaneous-connections-to-server filename(optional default=totransfer.tar)
Conducts an FTP transfer session.
Continuously repeats scenario (selected by editing the script) mixing ICMP Echo (ping), SNMP, NTP, DNS and FTP traffic (this last via ftpgen.py).
These utilities are used to generate traffic during the trafMon setup and configuration phase, or to qualify data links under “worst case” traffic patterns.
On the receiving side, following system daemons are “receiving” the traffic:
- “vsftpd” for the ftp traffic
- “ntpd“ for ntp traffic
- “snmpd“ for snmp
- “named“ for dns traffic
- “httpd“ for http traffic
5. The cron tables
A typical trafMon collector cron table, for unprivileged trafmon Linux account, includes:
- Execution every 10 minutes (or so) of the trafMon_loader.py, followed by trafMon_updateIpInfo.py
- Once per week, or less, trafMon_updateIpInfo.py –all
- The NetFlow data are extracted from the SiLK database once every hour (for the previous hour, when we can be pretty sure that all the connections are closed): trafMon_FormatNetFlow.py
- During a quite period at night, call the Aggr_activityvolumetable_first_level(“trafMon”, NULL, NULL), just after invocation of Update_ftp_data_in_ipcttable(“trafMon”, NULL, NULL) MySQL stored procedure.
- At another quiet time at night, optionally call the Aggr_activityvolumetable_netflow_first_level(“trafMon”, NULL, NULL) MySQL stored procedure.
- The Drop_procedure() is called regularly to clean the database of temporary tables (typically, tables created when generating reports).
- The schedule for trafMon_volumeReportGen.py and/or trafMon_detailReportGen.py scripts can be added to generate reports automatically.
- After careful monitoring of the growing rate of fine-grain trafMon raw observations tables, specific invocations of the Partition_drop() MySQL stored procedure can be automatically scheduled.
- Every minute, the trafMon_serverResurrect.sh /var/log/trafMon/resurrect.log “/opt/trafMon/bin/tmon_collector TMonServer” will permit to (re-)start a missing trafMon collector process.
A typical cron table for the privileged root Linux account, would include:
- Every minute, the trafMon_serverResurrect.sh (for mysqld and tomcat services only) or trafMon_probeResurrect.sh (unless invoked by unprivileged trafmon Linux account, thanks to setcap tuning).
- Every hour, the logrotate for the trafMon specific logrotate configuration
6. Optional CERT® SiLK
The SiLK Netflow receiver and database has three main configuration files, located in /var/silk and /usr/local/etc/:
- rwflowpack.conf: defines how the software itself should behave: where to log events, where to store data, etc.
- twoway-silk.conf and sensor.conf: defines which version of NetFlow the sensor(s) will receive, whether it will be over TCP or UDP and how to categorize the IPs.
These configuration files will be set up as needed or can be used as is.
If any other information is needed, please refer to the SiLK documentation.
The specific setup of a running MySQL server for its use by the trafMon tool consists of 3 steps:
- Defining two database users (e.g. tmon_db and tmon_birt: db_users_sample.sql); the first with full privileges over the trafMon databases, including bulk loading from files, and the second with limited right to query (read-only) and to execute pre-defined stored procedures. The user names and their encoded passwords must be configured in several (protected) files (/etc/trafMon/cred/db.cred, /var/www/html/trafMon/php/include.php, /var/lib/tomcat/webapps/birt/trafMon_reports/Library/trafMonDb.rptlibrary);
- Create and configure the file /etc/trafMon/ipInfo.ini with Activity/Location of all known IP address ranges;
- Executing the SQL script that creates and populates the (nearly empty) database with template schemas for temporary and persistent trafMon data tables and with the definition of all trafMon stored procedures: trafMon_templateDB.sql
Maybe the table trafMon_template.protocol_table needs to be tuned with service names and precedence assigned to TCP or UDP port numbers.
At first run of the trafMon_loader.py, the runtime database will be automatically created and initialised.
For all details, please refer to the trafMon Configuration and Administration Guide: trafMon v1.0.0 Configuration and Administration Guide, Oct 2020