NetFlow export
Preparation
First, restore T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins:
t2build -e -y
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? yes
Plugin folder emptied
Then compile the core (tranalyzer2) and the following plugins:
t2build tranalyzer2 basicFlow basicStats tcpStates tcpFlags macRecorder netflowSink
...
BUILD SUCCESSFUL
If you did not create a separate data and results directory yet, please do it now in another bash window, that facilitates your workflow:
mkdir ~/data ~/results
The sample PCAP used in this tutorial can be downloaded here: faf-exercise.pcap.
Please save it in your ~/data folder.
Now you are all set!
Dependencies
Note that the netflowSink plugin requires the basicFlow, basicStats, tcpStates and tcpFlags plugins. The macRecorder plugin is optional, but recommended.
As you can see we do not have the txtSink plugin, so no flow file is generated. If you wish to do that, you can add any sink you deem necessary. Nevertheless, any additional medium adds delays, which is crucial if you sniff from an interface.
To illustrate the configuration and application of the NetFlow export look at netflowSink and
log into nfcapd, so that the well known nfdump tool can read it.
If you want to benefit from Tranalyzer extended capabilities you can also use the socketSink plugin which sends flows to any location you deem appropriate.
Plugins and configuration
Let’s have a look at the configuration
netflowSink
vi src/netflowSink.h
...
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */
#define NF_SOCKTYPE 0 // Socket type: 0: UDP, 1: TCP
#define NF_VER 9 // NetFlow version 9 or 10 (IPFIX)
/* +++++++++++++++++++++ ENV / RUNTIME - conf Variables +++++++++++++++++++++ */
#define NF_NUM4FLWS 200 // Max # of IPv4 flows in one NetFlow message
#define NF_NUM6FLWS 100 // Max # of IPv6 flows in one NetFlow message
#define NF_SERVADD "127.0.0.1" // Destination address
#define NF_DPORT 9995 // Destination port
/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...The default address to log is the local interface, if you want to log remotely
change the address in NF_SERVADD.
The destination port is set to the nfcapd default, you can choose any convenient port.
Socket type is UDP, but you may change it, if you have a tool which requires it.
Choose the NetFlow version you want to export and set the maximum IPv4/6 flows
bunches which will be transport in one NetFlow message to the receiving server.
The default values work fine, but you may change them to optimize performance.
In that case, recompile netflowSink
In order to collect T2 flow data iinvoke nfcapd in another window of your local machine or on the
remote server, use the following command:
nfcapd -T all -B 1000000 -n sourcename,127.0.0.1,.
Or
nfcapd -T all -B 1000000 -n sourcename,serveraddress,.
We use increased buffering, so that nfcapd can keep up with Tranalyzer.
The sourcename identifies the sensor associated by the server address.
The . denotes the base directory where to save the compressed flows.
Now go back to the Tranalyzer window and start T2 with faf-exercise.pcap.
t2 -r ~/data/faf-exercise.pcap
Or from an interface: (Note to enable the inputbuffer if you expect hight traffic load. T2 i/O buffer is then threaded, and independent of the linux buffer, so performance is higher)
t2conf tranalyzer2 -D IO_BUFFERING=1 && t2build -R
st2 -i interface
[sudo] password for wurst:
...The file which is then produced by nfcapd: nfcapd.2019xxxxxxxx can be interpreted by nfdump. Here is a sample of an output
from nfdump. Just run your own traffic and play around with it.
nfdump -r nfcapd.2019xxxxxxxx -o extended -s srcip -s ip/flows -s dstport/pps/packets/bytes -s record/bytes
Aggregated flows 72
Top 10 flows ordered by bytes:
Date first seen Duration Proto Src IP Addr:Port Dst IP Addr:Port Flags Tos Packets Bytes pps bps Bpp Flows
2009-11-19 02:29:23.449 21.941 TCP 8fa6:b0a::.64334 -> c0a8:169::.49330 ...... 0 3101 4.3 M 141 1.6 M 1376 1
2009-11-19 00:37:24.836 0.765 TCP c6bd:ff4b::.80 -> c0a8:168::.1908 0xae 45 74 97078 96 1.0 M 1311 1
2009-11-18 12:36:56.878 0.093 TCP c6bd:ff4b::.80 -> c0a8:168::.1260 ...... 0 73 95603 784 8.2 M 1309 1
2009-11-18 12:36:57.319 0.057 TCP c6bd:ff4b::.80 -> c0a8:168::.1262 0xff 200 26 30820 456 4.3 M 1185 1
2009-11-19 00:37:25.961 0.049 TCP c6bd:ff4b::.80 -> c0a8:168::.1910 ...... 0 19 21634 387 3.5 M 1138 1
2009-11-18 17:41:07.717 41.899 TCP 3ff5:dd0b::.80 -> c0a8:168::.1379 0xf8 174 18 15606 0 2979 867 1
2009-11-18 23:33:34.260 1.025 TCP c0a8:166::.1911 -> c0a8:101::.25 0xab 192 20 7079 19 55250 353 1
2009-11-18 19:01:29.224 0.022 TCP c0a8:166::.1405 -> c0a8:101::.25 ...... 0 19 5796 863 2.1 M 305 1
2009-11-18 23:02:37.550 0.771 TCP c0a8:167::.1934 -> c0a8:101::.25 ..PR.F 25 18 5732 23 59476 318 1
2009-11-18 18:23:50.267 0.116 TCP c0a8:167::.1749 -> c0a8:101::.25 ...... 0 19 5679 163 391655 298 1
Top 10 Src IP Addr ordered by -:
Date first seen Duration Proto Src IP Addr Flows(%) Packets(%) Bytes(%) pps bps bpp
2009-11-18 12:36:55.001 43231.002 any c0a8:168:: 16(22.2) 148( 2.5) 8203( 0.2) 0 1 55
2009-11-19 02:29:22.972 328.674 any 8fa6:b0a:: 2( 2.8) 3112(52.7) 4.3 M(91.8) 9 103934 1372
2009-11-18 18:58:29.346 16505.939 any c0a8:166:: 5( 6.9) 89( 1.5) 22018( 0.5) 0 10 247
2009-11-18 18:37:00.264 28370.714 any c0a8:169:: 8(11.1) 1626(27.5) 21396( 0.5) 0 6 13
2009-11-18 17:59:33.904 20041.382 any c0a8:101:: 21(29.2) 551( 9.3) 24948( 0.5) 0 9 45
2009-11-18 17:59:33.904 18534.881 any c0a8:167:: 7( 9.7) 123( 2.1) 27914( 0.6) 0 12 226
2009-11-18 17:41:07.717 41.899 any 3ff5:dd0b:: 2( 2.8) 26( 0.4) 19790( 0.4) 0 3778 761
2009-11-18 12:36:56.878 43229.132 any c6bd:ff4b:: 6( 8.3) 204( 3.5) 252093( 5.4) 0 46 1235
2009-11-18 12:36:55.165 3.092 any 4d43:2cce:: 5( 6.9) 23( 0.4) 5403( 0.1) 7 13979 234
Top 10 IP Addr ordered by flows:
Date first seen Duration Proto IP Addr Flows(%) Packets(%) Bytes(%) pps bps bpp
2009-11-18 17:59:33.904 20041.382 any c0a8:101:: 42(58.3) 894(15.1) 98540( 2.1) 0 39 110
2009-11-18 12:36:55.001 43231.009 any c0a8:168:: 32(44.4) 452( 7.7) 286413( 6.2) 0 53 633
2009-11-18 18:37:00.264 28671.382 any c0a8:169:: 16(22.2) 4906(83.1) 4.3 M(92.4) 0 1199 876
2009-11-18 17:59:33.904 18534.881 any c0a8:167:: 14(19.4) 318( 5.4) 37644( 0.8) 0 16 118
2009-11-18 12:36:56.871 43229.139 any c6bd:ff4b:: 12(16.7) 275( 4.7) 254008( 5.5) 0 47 923
2009-11-18 12:36:55.001 3.256 any 4d43:2cce:: 10(13.9) 49( 0.8) 7058( 0.2) 15 17341 144
2009-11-18 18:58:29.346 16505.940 any c0a8:166:: 10(13.9) 226( 3.8) 28130( 0.6) 0 13 124
2009-11-18 17:41:07.711 41.905 any 3ff5:dd0b:: 4( 5.6) 47( 0.8) 22067( 0.5) 1 4212 469
2009-11-19 02:29:22.891 328.755 any 8fa6:b0a:: 4( 5.6) 4637(78.6) 4.3 M(91.8) 14 103911 920
Top 10 Dst Port ordered by packets:
Date first seen Duration Proto Dst Port Flows(%) Packets(%) Bytes(%) pps bps bpp
2009-11-19 02:29:23.449 21.941 any 49330 1( 1.4) 3101(52.5) 4.3 M(91.8) 141 1.6 M 1376
2009-11-19 02:29:23.371 27.607 any 64334 1( 1.4) 1514(25.7) 0( 0.0) 54 0 0
2009-11-18 17:59:33.904 20041.381 any 25 21(29.2) 343( 5.8) 73592( 1.6) 0 29 214
2009-11-18 12:36:55.001 43231.002 any 80 13(18.1) 118( 2.0) 5847( 0.1) 0 1 49
2009-11-19 00:37:24.836 0.765 any 1908 1( 1.4) 74( 1.3) 97078( 2.1) 96 1.0 M 1311
2009-11-18 12:36:56.878 0.093 any 1260 1( 1.4) 73( 1.2) 95603( 2.1) 784 8.2 M 1309
2009-11-18 18:38:00.152 0.023 any 49219 1( 1.4) 30( 0.5) 799( 0.0) 1304 277913 26
2009-11-18 18:23:50.267 0.116 any 1749 1( 1.4) 30( 0.5) 1405( 0.0) 258 96896 46
2009-11-18 17:59:33.904 2.652 any 1397 1( 1.4) 30( 0.5) 1440( 0.0) 11 4343 48
2009-11-18 18:37:00.264 1.597 any 49218 1( 1.4) 29( 0.5) 1519( 0.0) 18 7609 52
Top 10 Dst Port ordered by bytes:
Date first seen Duration Proto Dst Port Flows(%) Packets(%) Bytes(%) pps bps bpp
2009-11-19 02:29:23.449 21.941 any 49330 1( 1.4) 3101(52.5) 4.3 M(91.8) 141 1.6 M 1376
2009-11-19 00:37:24.836 0.765 any 1908 1( 1.4) 74( 1.3) 97078( 2.1) 96 1.0 M 1311
2009-11-18 12:36:56.878 0.093 any 1260 1( 1.4) 73( 1.2) 95603( 2.1) 784 8.2 M 1309
2009-11-18 17:59:33.904 20041.381 any 25 21(29.2) 343( 5.8) 73592( 1.6) 0 29 214
2009-11-18 12:36:57.319 0.057 any 1262 1( 1.4) 26( 0.4) 30820( 0.7) 456 4.3 M 1185
2009-11-19 00:37:25.961 0.049 any 1910 1( 1.4) 19( 0.3) 21634( 0.5) 387 3.5 M 1138
2009-11-18 17:41:07.717 41.899 any 1379 1( 1.4) 18( 0.3) 15606( 0.3) 0 2979 867
2009-11-18 12:36:55.001 43231.002 any 80 13(18.1) 118( 2.0) 5847( 0.1) 0 1 49
2009-11-18 12:36:57.725 0.028 any 1264 1( 1.4) 7( 0.1) 5268( 0.1) 250 1.5 M 752
2009-11-18 17:41:18.229 31.387 any 1384 1( 1.4) 8( 0.1) 4184( 0.1) 0 1066 523
Top 10 Dst Port ordered by pps:
Date first seen Duration Proto Dst Port Flows(%) Packets(%) Bytes(%) pps bps bpp
2009-11-18 18:38:00.152 0.023 any 49219 1( 1.4) 30( 0.5) 799( 0.0) 1304 277913 26
2009-11-18 19:01:29.224 0.022 any 1405 1( 1.4) 28( 0.5) 632( 0.0) 1272 229818 22
2009-11-18 19:25:38.071 0.033 any 49561 1( 1.4) 28( 0.5) 1466( 0.0) 848 355393 52
2009-11-18 18:42:03.669 0.033 any 1806 1( 1.4) 28( 0.5) 1370( 0.0) 848 332121 48
2009-11-18 19:14:19.091 0.032 any 1836 1( 1.4) 26( 0.4) 1370( 0.0) 812 342500 52
2009-11-18 12:36:56.878 0.093 any 1260 1( 1.4) 73( 1.2) 95603( 2.1) 784 8.2 M 1309
2009-11-18 12:36:57.319 0.057 any 1262 1( 1.4) 26( 0.4) 30820( 0.7) 456 4.3 M 1185
2009-11-19 00:37:25.961 0.049 any 1910 1( 1.4) 19( 0.3) 21634( 0.5) 387 3.5 M 1138
2009-11-18 18:23:50.267 0.116 any 1749 1( 1.4) 30( 0.5) 1405( 0.0) 258 96896 46
2009-11-18 12:36:57.725 0.028 any 1264 1( 1.4) 7( 0.1) 5268( 0.1) 250 1.5 M 752
Summary: total flows: 72, total bytes: 4651854, total packets: 5902, avg bps: 740, avg pps: 0, avg bpp: 788
Time window: 2009-11-18 12:36:55 - 2009-11-19 02:34:51
Total flows processed: 72, Blocks skipped: 0, Bytes read: 9928
Sys: 0.003s flows/second: 22584.7 Wall: 0.000s flows/second: 75550.9Conclusion
For more info about nfdump, here is a tutorial.
Note, that Tranalyzer in combination with TAWK provides much more flexibility, especially when it comes to non standard questions,
as in troubleshooting or traffic mining. nfcapd can also be emulated by the socketSink plugin and netcat in an easy way.