Flow mode

aggregation flow FDLSFINDEX FDURLIMIT FLOW_TIMEOUT

Introduction

This tutorial details the use of the Flow mode being the default operational technique of the Anteater. As we discuss the enhanced aggregation and masking features in a different tutorial, we like to concentrate here on the flow timeout features often used in anomaly research, such as botnet detection and practical network monitoring applications.

Preparation

In order to do so, we need to prepare T2. If you did not complete the tutorials before, just follow the procedure described below.

First, restore T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins and compile only basicFlow, basicStats and txtSink.

t2build -e -y

Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? yes
Plugin folder emptied
t2build tranalyzer2 basicFlow basicStats txtSink

...
BUILD SUCCESSFUL

If you did not create a separate data and results directory yet, please do it now in another cmd window, that facilitates your workflow:

mkdir ~/data ~/results

The anonymized sample PCAP being used here, can be downloaded here: faf-exercise.pcap. Please extract it under your ~/data folder. Now you are all set for T2 flow timeout experiments.

Flow timeout control

The Flow timeout controls of the Anteater is a core function, hence residing in the tranalyzer.h file:

tranalyzer2

vi src/tranalyzer.h

...
// Time mode
#define RELTIME 0 // 0: Absolute time, 1: Relative internal time

// Maximum lifetime of a flow
#define FDURLIMIT 0  // if > 0; forced flow life span of n +- 1 seconds
#define FDLSFINDEX 0 // if (FDURLIMIT) 0: Different findex; 1: Same findex for flows of a superflow

// The standard timeout for a flow in seconds
#define FLOW_TIMEOUT 182 // flow timeout after a packet is not seen after n seconds
...

RELTIME defines the internal time base to be absolute or relative to the pcap or interface ops start; default is absolute. For graphical applications, it is sometimes convenient to have smaller time values relative to 0. We used it in the Basic traffic volume and connection analysis tutorial to produce a anomaly plot for the connStat plugin.

FLOW_TIMEOUT controls the timeout and releases flows from memory if no more packets are seen since FLOW_TIMEOUT seconds.

It is chosen a bit larger than three minutes, as most L7 protocols have lower timeouts, so we capture most of them into one flow. Nevertheless, a large FLOW_TIMEOUT saves flow allocation and output operations, but also increases the flow lifetime in memory. So you need to optimize this value in high speed applications.

FDURLIMIT controls independent of FLOW_TIMEOUT the forced release of each flow in memory, thus being a simulated Cisco flow mode. It is activated with a non zero value in seconds. Hence, a value of 10 would terminate an active flow after 10 seconds if a new packet is encountered, and start a new sub-flow with the next packet having the same flow hash. If there is no new packet with the right hash value, then there is no flow release, until an appropriate packet arrives.

This is very different from FLOW_TIMEOUT because any packet can trigger the flow termination.

It is a built in FORCE MODE as it does not use the T2_RM_FLOW(flowP) function, see Force mode tutorial.

If FDURLIMIT is enabled FDLSFINDEX controls the enumeration of the sub-flows. Set it to 1 and all sub-flows keep the findex of the original flow. Otherwise the findex increments normally. This feature facilitates to extract sub-flows belonging to a flow.

t2 -r ~/data/faf-exercise.pcap -w ~/results/

================================================================================
Tranalyzer 0.8.14 (Anteater), Tarantula. PID: 45166
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.14
    02: basicStats, 0.8.14
    03: txtSink, 0.8.14
[INF] IPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 406105 (406.11 K)
[INF] IPv6 Ver: 5, Rev: 17122020, Range Mode: 0, subnet ranges loaded: 51345 (51.34 K)
Processing file: /home/wurst/data/faf-exercise.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1258544215.037210 sec (Wed 18 Nov 2009 11:36:55 GMT)
Dump stop : 1258594491.683288 sec (Thu 19 Nov 2009 01:34:51 GMT)
Total dump duration: 50276.646078 sec (13h 57m 56s)
Finished processing. Elapsed time: 0.037299 sec
Finished unloading flow memory. Time: 0.037335 sec
Percentage completed: 100.00%
Number of processed packets: 5902 (5.90 K)
Number of processed bytes: 4993414 (4.99 M)
Number of raw bytes: 4993414 (4.99 M)
Number of pcap bytes: 5087870 (5.09 M)
Number of IPv4 packets: 5902 (5.90 K) [100.00%]
Number of A packets: 1986 (1.99 K) [33.65%]
Number of B packets: 3916 (3.92 K) [66.35%]
Number of A bytes: 209315 (209.31 K) [4.19%]
Number of B bytes: 4784099 (4.78 M) [95.81%]
Average A packet load: 105.40
Average B packet load: 1221.68 (1.22 K)
--------------------------------------------------------------------------------
basicStats: Biggest L3 talker: 143.166.11.10 (US): 3101 (3.10 K) [52.54%] packets
basicStats: Biggest L3 talker: 143.166.11.10 (US): 4436320 (4.44 M) [88.84%] bytes
--------------------------------------------------------------------------------
Headers count: min: 3, max: 3, average: 3.00
Number of TCP packets: 5902 (5.90 K) [100.00%]
Number of TCP bytes: 4993414 (4.99 M) [100.00%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 73
Number of processed A flows: 37 [50.68%]
Number of processed B flows: 36 [49.32%]
Number of request     flows: 36 [49.32%]
Number of reply       flows: 37 [50.68%]
Total   A/B    flow asymmetry: 0.01
Total req/rply flow asymmetry: -0.01
Number of processed   packets/flows: 80.85
Number of processed A packets/flows: 53.68
Number of processed B packets/flows: 108.78
Number of processed total packets/s: 0.12
Number of processed A+B   packets/s: 0.12
Number of processed A     packets/s: 0.04
Number of processed   B   packets/s: 0.08
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.00
Average full raw bandwidth: 795 b/s
Average full bandwidth : 792 b/s
Max number of flows in memory: 18 [0.01%]
Memory usage: 0.01 GB [0.02%]
Aggregated flowStat=0x0400000000004000
[INF] IPv4 flows

The end report states that there are 73 flows in total and 1986 A flow packets and 3916 B flow packets. Now select a flow which is long enough to be early terminated, such as the one with port 1379 involved.

tawk 'port(1379)' ~/results/faf-exercise_flows.txt | tcol

%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  vlanID  srcIP          srcIPCC  srcIPOrg               srcPort  dstIP          dstIPCC  dstIPOrg               dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT    stdIAT    pktps      bytps     pktAsm  bytAsm
A     10       0x0400000000004000  1258562467.749142  1258562509.633370  41.884228  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800           192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        14          18           1801         15606         0         455       128.6429    183.7502    0       22.97809  2.991731  6.053773  0.3342547  42.99948  -0.125  -0.7930717
B     10       0x0400000000004001  1258562467.754689  1258562509.653962  41.899273  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800           63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        18          14           15606        1801          0         1380      867         568.1608    0       22.97829  2.327737  5.515566  0.4296017  372.4647  0.125   0.7930717

Now let us forcefully terminate flows above a certain length in the next chapter.

Flow duration limit: FDURLIMIT

Set FDURLIMIT to 3, recompile and rerun T2 on the pcap. Then t2 will terminate every flow with the time frame >= 3 seconds and restart a new flow with the same hash but different findex.

t2conf tranalyzer2 -D FDURLIMIT=3 && t2build tranalyzer2

t2 -r ~/data/faf-exercise.pcap -w ~/results/faf-exercise3

================================================================================
Tranalyzer 0.8.14 (Anteater), Tarantula. PID: 46212
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.14
    02: basicStats, 0.8.14
    03: txtSink, 0.8.14
[INF] IPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 406105 (406.11 K)
[INF] IPv6 Ver: 5, Rev: 17122020, Range Mode: 0, subnet ranges loaded: 51345 (51.34 K)
Processing file: /home/wurst/data/faf-exercise.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1258544215.037210 sec (Wed 18 Nov 2009 11:36:55 GMT)
Dump stop : 1258594491.683288 sec (Thu 19 Nov 2009 01:34:51 GMT)
Total dump duration: 50276.646078 sec (13h 57m 56s)
Finished processing. Elapsed time: 0.005011 sec
Finished unloading flow memory. Time: 0.005026 sec
Percentage completed: 100.00%
Number of processed packets: 5902 (5.90 K)
Number of processed bytes: 4993414 (4.99 M)
Number of raw bytes: 4993414 (4.99 M)
Number of pcap bytes: 5087870 (5.09 M)
Number of IPv4 packets: 5902 (5.90 K) [100.00%]
Number of A packets: 1986 (1.99 K) [33.65%]
Number of B packets: 3916 (3.92 K) [66.35%]
Number of A bytes: 209315 (209.31 K) [4.19%]
Number of B bytes: 4784099 (4.78 M) [95.81%]
Average A packet load: 105.40
Average B packet load: 1221.68 (1.22 K)
--------------------------------------------------------------------------------
basicStats: Biggest L3 talker: 143.166.11.10 (US): 558 [9.45%] packets
basicStats: Biggest L3 talker: 143.166.11.10 (US): 799321 (799.32 K) [16.01%] bytes
--------------------------------------------------------------------------------
Headers count: min: 3, max: 3, average: 3.00
Number of TCP packets: 5902 (5.90 K) [100.00%]
Number of TCP bytes: 4993414 (4.99 M) [100.00%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 104
Number of processed A flows: 53 [50.96%]
Number of processed B flows: 51 [49.04%]
Number of request     flows: 52 [50.00%]
Number of reply       flows: 52 [50.00%]
Total   A/B    flow asymmetry: 0.02
Total req/rply flow asymmetry: 0.00
Number of processed   packets/flows: 56.75
Number of processed A packets/flows: 37.47
Number of processed B packets/flows: 76.78
Number of processed total packets/s: 0.12
Number of processed A+B   packets/s: 0.12
Number of processed A     packets/s: 0.04
Number of processed   B   packets/s: 0.08
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.00
Average full raw bandwidth: 795 b/s
Average full bandwidth : 792 b/s
Max number of flows in memory: 18 [0.01%]
Memory usage: 0.01 GB [0.02%]
Aggregated flowStat=0x0400020000004001
[INF] IPv4 flows

As expected we have now more flows, but the same amount of packets, otherwise something would be very wrong. The reporting of basicStats changed as the packets of an specific IP are now scattered over many flows. And the end report has the RMFLOW bit set to indicate that flows are forcefully terminated.

If you extract now the same flow as above you realize that the flows have a smaller length as three seconds. Note the time difference between timeLast of flow 10 and timeFirst of flow 11 of 10 seconds. Hence, after 10 seconds a new packet came in and flow 10 terminated with its last packet at 1258562467.900050 and restarted B-flow 11 with this very packet at 1258562477.772691. This happens when you have bursty traffic. If the traffic would be regular, all flows would have a duration of three seconds. The RMFLOW bit indicates the flow where the FDURLIMIT timeout occurred.

tawk 'port(1379)' ~/results/faf-exercise3_flows.txt | tcol

%dir  flowInd  flowStat            timeFirst          timeLast           duration  numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  vlanID  srcIP          srcIPCC  srcIPOrg               srcPort  dstIP          dstIPCC  dstIPOrg               dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT       stdIAT       pktps     bytps     pktAsm      bytAsm
A     10       0x0400020000004000  1258562467.749142  1258562467.900050  0.150908  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800           192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        4           3            449          444           0         449       112.25      159.8441    0       0.144417  0.037727     0.05336916   26.50622  2975.323  0.1428571   0.005599104
B     10       0x0400000000004001  1258562467.754689  1258562467.761692  0.007003  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800           63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        3           4            444          449           0         444       148         170.8957    0       0.006751  0.002334333  0.002289841  428.3878  63401.4   -0.1428571  -0.005599104
A     11       0x0400000000004000  1258562477.713894  1258562478.454425  0.740531  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800           192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        8           13           1352         15162         0         455       169         185.5386    0       0.380024  0.09256638   0.09487616   10.80306  1825.717  -0.2380952  -0.8362601
B     11       0x0400020000004001  1258562477.772691  1258562478.454017  0.681326  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800           63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        13          8            15162        1352          10        1380      1166.308    353.6241    0       0.426916  0.05240969   0.0970576    19.08044  22253.67  0.2380952   0.8362601
A     13       0x0400020000004000  1258562501.432514  1258562501.432514  0.000000  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800           192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
B     13       0x0400000000004001  1258562501.432305  1258562501.432305  0.000000  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800           63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
A     16       0x0400000000004000  1258562509.633370  1258562509.633370  0.000000  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800           192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
B     16       0x0400000000004001  1258562509.653962  1258562509.653962  0.000000  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800           63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0

Yes I picked this traffic on purpose to demonstrate that behavior of T2. As you can see our flow 10 is now divided into four flows 10, 11, 13 and 16. But if you want to select all the sub flows you need to select the port or the full 6 tuple if there are many more flows this the same port or IP pair. In order to facilitate that operation, since 0.8.8lmw2 the FDLSFINEX switch was added to label all subflows with the same findex.

Flow duration, same index: FDLSFINDEX

In order to label all sub-flows with the same findex, the constant FDLSFINDEX has to be enabled in conjunction with a non-zero value of FDURLIMIT in tranalyzer.h or just use t2conf as indicated below.

t2conf tranalyzer2 -D FDLSFINDEX=1 && t2build tranalyzer2

t2 -r ~/data/faf-exercise.pcap -w ~/results/faf-exercise3f

...
--------------------------------------------------------------------------------
basicStats: Biggest L3 talker: 143.166.11.10 (US): 558 [9.45%] packets
basicStats: Biggest L3 talker: 192.168.1.104: 7366431884 (7.37 G) [147522.95%] bytes
--------------------------------------------------------------------------------
Headers count: min: 3, max: 3, average: 3.00
Number of GRE packets: 10 [0.17%]
Number of TCP packets: 5902 (5.90 K) [100.00%]
Number of TCP bytes: 4993414 (4.99 M) [100.00%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...

The end report looks the same as above with the exception that now the FDLSIDX bit is set, indicating that all sub-flows belonging to the original, un-terminated flow have now the same findex. Now select flow 10 again and you discover that the flows 10, 11, 13 and 16 have now all findex 10 which is much easier to extract as indicated below.

tawk 'flow(10)' ~/results/faf-exercise3f_flows.txt | tcol

%dir  flowInd  flowStat            timeFirst          timeLast           duration  numHdrDesc  numHdrs  hdrDesc  srcMac                                                 dstMac                                                 ethType  vlanID  srcIP          srcIPCC  srcIPOrg               srcPort  dstIP          dstIPCC  dstIPOrg               dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT       stdIAT       pktps     bytps     pktAsm      bytAsm
A     10       0x0400020000004000  1258562467.749142  1258562467.900050  0.150908  1           0                 fe:63:a3:b6:0d:22;46:d9:fe:63:a3:b6;ff:44:23:f6:80:ed  6d:59:a5:aa:46:d9;b1:e7:6d:59:a5:aa;0f:da:77:2f:6d:62  0x0800           192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        4           3            537          512           20        469       134.25      158.9655    0       0.144417  0.037727     0.05336916   26.50622  3558.46   0.1428571   0.02383222
B     10       0x0400000000004001  1258562467.754689  1258562467.761692  0.007003  1           0                 a5:aa:46:d9:fe:63;68:75:6e:64:65:72;42:3e:be:0f:da:77  eb:f0:b1:e7:6d:59;47:45:54:20:2f:74;65:0d:0a:0d:0a:45  0x0800           63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        3           4            512          537           20        464       170.6667    169.3718    0       0.006751  0.002334333  0.002289841  428.3878  73111.52  -0.1428571  -0.02383222
A     10       0x4400000000004000  1258562477.713894  1258562478.454425  0.740531  1           0                 53:92:60:01:48:7c;0a:53:65:72:76:65;0e:29:7b:63:67:d3  ff:44:23:f6:80:ed;30:30:20:4f:4b:0d;f4:37:fa:7b:fd:92  0x0800           192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        8           13           1512         15422         20        475       189         185.5386    0       0.380024  0.09256638   0.09487616   10.80306  2041.778  -0.2380952  -0.8214244
B     10       0x4400020000004001  1258562477.772691  1258562478.454017  0.681326  1           0                 00:00:00:00:00:00;0a:53:65:72:76:65;ac:fc:04:db:61:d4  00:00:00:00:00:00;30:30:20:4f:4b:0d;56:29:14:c7:fc:a2  0x0800           63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        13          8            15422        1512          30        1400      1186.308    353.6241    0       0.426916  0.05240969   0.0970576    19.08044  22635.27  0.2380952   0.8214244
A     10       0x4400020000004000  1258562501.432514  1258562501.432514  0.000000  1           1                 8d:26:6f:2f:b2:39                                      b2:87:64:65:6c:99                                      0x0800           192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        1           1            20           20            20        20        20          0           0       0         0            0            0         0         0           0
B     10       0x4400000000004001  1258562501.432305  1258562501.432305  0.000000  1           8                 36:11:6e:37:b2:87                                      00:00:00:00:00:00                                      0x0800           63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        1           1            20           20            20        20        20          0           0       0         0            0            0         0         0           0
A     10       0x4400000000004000  1258562509.633370  1258562509.633370  0.000000  1           1                 8d:26:6f:2f:b2:39                                      b2:87:64:65:6c:99                                      0x0800           192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        1           1            20           20            20        20        20          0           0       0         0            0            0         0         0           0
B     10       0x4400000000004001  1258562509.653962  1258562509.653962  0.000000  1           8                 5e:ce:75:33:b2:87                                      00:00:00:00:00:00                                      0x0800           63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        1           1            20           20            20        20        20          0           0       0         0            0            0         0         0           0

Now let’s explore the difference to the flow timeout.

No packets seen since: FLOW_TIMEOUT

As explained above the constant FLOW_TIMEOUT controls the lifetime of a flow if a packet was not seen. Which is very different from the FDURLIMIT which terminates a flow and starts a new flow after an internal flow life time.

The timeout of flows can be very useful if you want to follow protocols and change the timeout in dependence of a protocol state, as implemented in tcpStates plugin.

FLOW_TIMEOUT is actually a float, so you can set a timeout Π seconds if you like. We choose 3 as we like to compare it to the previous experiments. So switch FDURLIMIT off, set FLOW_TIMEOUT to 3, recompile and rerun t2 on the same pcap.

t2conf tranalyzer2 -D FDURLIMIT=0 -D FLOW_TIMEOUT=3 && t2build tranalyzer2

t2 -r ~/data/faf-exercise.pcap -w ~/results/faf-exercise3t

...
--------------------------------------------------------------------------------
basicStats: Biggest L3 talker: 143.166.11.10 (US): 3101 (3.10 K) [52.54%] packets
basicStats: Biggest L3 talker: 143.166.11.10 (US): 4436320 (4.44 M) [88.84%] bytes
--------------------------------------------------------------------------------
Headers count: min: 3, max: 3, average: 3.00
Number of TCP packets: 5902 (5.90 K) [100.00%]
Number of TCP bytes: 4993414 (4.99 M) [100.00%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 88
Number of processed A flows: 45 [51.14%]
Number of processed B flows: 43 [48.86%]
Number of request     flows: 44 [50.00%]
Number of reply       flows: 44 [50.00%]
Total   A/B    flow asymmetry: 0.02
Total req/rply flow asymmetry: 0.00
Number of processed   packets/flows: 67.07
Number of processed A packets/flows: 44.13
Number of processed B packets/flows: 91.07
Number of processed total packets/s: 0.12
Number of processed A+B   packets/s: 0.12
Number of processed A     packets/s: 0.04
Number of processed   B   packets/s: 0.08
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...

Now we have 88 flows, not 73 as with 182 sec timeout or 104 with FDURLIMIT. What is the reason? If you select the same port 1379 as before, you discover that flow 10 shows the same behavior as with FDURLIMIT set to 3 sec, because all flows did not see a packet for 3 seconds, hence they terminate and the following packet landed in the next flow. Oups, bad example to show the difference? Actually not, because it illustrates the need to be observant, because the circumstances of the traffic produce for the this very flow the same flow chunks.

tawk 'port(1379)' ~/results/faf-exercise3t_flows.txt | tcol

%dir  flowInd  flowStat            timeFirst          timeLast           duration  numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  vlanID  srcIP          srcIPCC  srcIPOrg               srcPort  dstIP          dstIPCC  dstIPOrg               dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT       stdIAT       pktps     bytps     pktAsm      bytAsm
A     10       0x0400000000004000  1258562467.749142  1258562467.900050  0.150908  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800           192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        4           3            449          444           0         449       112.25      159.8441    0       0.144417  0.037727     0.05336916   26.50622  2975.323  0.1428571   0.005599104
B     10       0x0400000000004001  1258562467.754689  1258562467.761692  0.007003  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800           63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        3           4            444          449           0         444       148         170.8957    0       0.006751  0.002334333  0.002289841  428.3878  63401.4   -0.1428571  -0.005599104
A     11       0x0400000000004000  1258562477.713894  1258562478.454425  0.740531  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800           192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        8           13           1352         15162         0         455       169         185.5386    0       0.380024  0.09256638   0.09487616   10.80306  1825.717  -0.2380952  -0.8362601
B     11       0x0400000000004001  1258562477.772691  1258562478.454017  0.681326  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800           63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        13          8            15162        1352          10        1380      1166.308    353.6241    0       0.426916  0.05240969   0.0970576    19.08044  22253.67  0.2380952   0.8362601
A     13       0x0400000000004000  1258562501.432514  1258562501.432514  0.000000  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800           192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
B     13       0x0400000000004001  1258562501.432305  1258562501.432305  0.000000  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800           63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
A     16       0x0400000000004000  1258562509.633370  1258562509.633370  0.000000  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800           192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
B     16       0x0400000000004001  1258562509.653962  1258562509.653962  0.000000  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800           63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0$

Now look for port 49330, you will see two flows, one single A with one packet. The findex numbers differ from each other, therefore we select the port.

tawk 'port(49330)' ~/results/faf-exercise3t_flows.txt | tcol

%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  vlanID  srcIP          srcIPCC  srcIPOrg           srcPort  dstIP          dstIPCC  dstIPOrg           dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT       stdIAT      pktps     bytps     pktAsm      bytAsm
A     42       0x0400000000004000  1258594163.408285  1258594185.427968  22.019683  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800           192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        1513        3101         0            4268858       0         0         0           0           0       0.671332  0.01455366   0.0370894   68.71125  0         -0.3441699  -1
B     42       0x0400000000004001  1258594163.487027  1258594185.427506  21.940479  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800           143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        3101        1513         4268858      0             0         1380      1376.607    60.23097    0       0.67109   0.007075286  0.02745561  141.3369  194565.4  0.3441699   1
A     44       0x0400000000004000  1258594191.015208  1258594191.015208  0.000000   1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800           192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        1           0            0            0             0         0         0           0           0       0         0            0           0         0         1           0

The FLOW_TIMEOUT produces one more flow 44, a packet which comes after the timeout. Most of the packets are coming very regularly, in contrast to flow 10 in the example above.

The same flow from the previous chapter shows nine flows mostly evenly 3 seconds long. See the differences between those modes? Compare it to the same flow from the previous chapter listed below.

tawk 'port(49330)' ~/results/faf-exercise3f_flows.txt | tcol

%dir  flowInd  flowStat            timeFirst          timeLast           duration  numHdrDesc  numHdrs  hdrDesc  srcMac                                                 dstMac                                                 ethType  vlanID  srcIP          srcIPCC  srcIPOrg           srcPort  dstIP          dstIPCC  dstIPOrg           dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT       stdIAT      pktps     bytps     pktAsm      bytAsm
A     36       0x0400000000004000  1258594163.408285  1258594165.927703  2.519418  1           0                 6f:64:65:20:28:31;73:69:76:65:20:4d;ff:00:74:41:8d:45  73:69:76:65:20:4d;04:02:13:6f:4b:cd;09:80:bd:e9:fe:ff  0x0800           192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        64          103          2132         140070        20        48        33.3125     11.21809    0       0.671332  0.03936591   0.1132588   25.40269  846.2272  -0.2335329  -0.9700145
B     36       0x0400020000004001  1258594163.487027  1258594165.927248  2.440221  1           0                 4b:cd:73:69:76:65;20:b7:0d:5d:25:d7;6a:5d:b3:75:06:86  01:01:04:02:13:6f;40:36:58:4a:06:7c;00:00:00:00:00:1c  0x0800           143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        103         64           140070       2132          21        1400      1359.903    216.1209    0       0.67109   0.02369147   0.09572505  42.20929  57400.54  0.2335329   0.9700145
A     36       0x4400000000004000  1258594166.005912  1258594168.988792  2.982880  1           0                 35:90:67:71:15:c1;4b:34:e8:e4:32:c2;f1:a7:67:71:15:c1  72:2e:6c:a9:4a:b6;67:71:15:c1:5c:92;bd:62:dc:62:7d:17  0x0800           192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        223         444          5256         620220        20        40        23.56951    5.735339    0       0.079202  0.01337614   0.02821018  74.75996  1762.055  -0.3313343  -0.9831936
B     36       0x4400020000004001  1258594166.005251  1258594168.988590  2.983339  1           0                 20:b7:0d:5d:25:d7;64:8f:94:1e:39:1d;3d:a9:d9:9b:e3:70  40:36:58:4a:06:7c;86:94:0f:95:c7:74;d7:c2:b0:23:80:77  0x0800           143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        444         223          620220       5256          100       1400      1396.892    59.41461    0       0.079166  0.006719232  0.02111784  148.8265  207894.6  0.3313343   0.9831936
A     36       0x4400000000004000  1258594169.065239  1258594171.945675  2.880436  1           0                 db:7c:58:f8:e7:c6;1f:88:a4:9d:18:76;ff:25:95:84:69:f2  e2:a9:31:cb:0a:5f;9c:d9:2b:f5:73:55;29:cb:97:0c:ac:60  0x0800           192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        194         402          4564         561943        20        32        23.52577    5.416288    0       0.128673  0.01484761   0.03038867  67.35091  1584.482  -0.3489933  -0.9838872
B     36       0x4400020000004001  1258594169.064789  1258594171.944501  2.879712  1           0                 20:b7:0d:5d:25:d7;99:e1:f1:11:d1:9a                    40:36:58:4a:06:7c;74:97:7e:53:6b:82                    0x0800           143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        402         194          561943       4564          543       1400      1397.868    42.63219    0       0.128665  0.007163462  0.02259158  139.5973  195138.6  0.3489933   0.9838872
A     36       0x4400000000004000  1258594172.021878  1258594174.978021  2.956143  1           0                 a8:11:a9:8d:21:5a;86:cf:16:c4:b1:28;01:37:9b:6a:46:36  8d:81:3b:ac:da:18;79:36:3f:ef:b6:e5;c7:17:6f:fe:9f:05  0x0800           192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        226         436          5636         610215        20        48        24.93805    6.821356    0       0.124433  0.01308028   0.02880465  76.45097  1906.538  -0.3172205  -0.9816968
B     36       0x4400020000004001  1258594172.021456  1258594174.977086  2.955630  1           0                 20:b7:0d:5d:25:d7;29:2d:49:4b:5b:14                    40:36:58:4a:06:7c;ae:d8:e2:0a:d8:1a                    0x0800           143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        436         226          610215       5636          1215      1400      1399.576    8.828124    0       0.124927  0.006778968  0.02202431  147.5151  206458.5  0.3172205   0.9816968
A     36       0x4400000000004000  1258594175.055730  1258594177.968394  2.912664  1           0                 1f:93:d9:2e:17:f1;34:cc:ea:ef:04:61;51:c6:08:2a:ae:a5  37:64:68:30:f6:78;44:a4:74:6f:5f:42;79:8e:65:d3:d1:f1  0x0800           192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        233         527          5664         737042        20        48        24.30901    7.982298    0       0.079708  0.0125007    0.02766376  79.9955   1944.612  -0.3868421  -0.9847476
B     36       0x4400020000004001  1258594175.055030  1258594177.966989  2.911959  1           0                 20:b7:0d:5d:25:d7;bf:37:09:ad:90:41;35:51:96:70:6e:50  40:36:58:4a:06:7c;8f:cd:a5:bc:6d:02;97:e4:4a:25:76:c9  0x0800           143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        527         233          737042       5664          716       1400      1398.562    29.66002    0       0.07942   0.005525537  0.01959059  180.9778  253108.7  0.3868421   0.9847476
A     36       0x4400000000004000  1258594178.040853  1258594180.952029  2.911176  1           0                 22:30:af:3d:9e:1c;2a:f7:81:67:e2:47;e0:da:4a:86:94:77  51:a3:5b:b4:68:c1;4e:8b:e3:fe:a5:b2;9d:cd:c8:01:28:7f  0x0800           192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        261         558          6380         780349        20        40        24.44444    5.955973    0       0.078208  0.01115393   0.02653476  89.65449  2191.554  -0.3626374  -0.9837809
B     36       0x4400020000004001  1258594178.040152  1258594180.951598  2.911446  1           0                 20:b7:0d:5d:25:d7;ff:c9:2f:09:63:f2                    40:36:58:4a:06:7c;37:43:6e:ad:d6:f6                    0x0800           143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        558         261          780349       6380          549       1400      1398.475    35.87016    0       0.078343  0.005217645  0.01904833  191.6573  268028    0.3626374   0.9837809
A     36       0x4400000000004000  1258594181.028734  1258594183.937406  2.908672  1           0                 f5:69:e6:87:99:1e;70:ce:95:5d:ca:97;79:c7:a1:8e:1b:3a  48:ad:11:b0:73:08;ca:b6:29:8e:58:8f;74:6c:eb:d8:d9:71  0x0800           192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        221         476          5408         666282        20        48        24.47059    6.61699     0       0.079713  0.01316141   0.02855756  75.97969  1859.268  -0.3658537  -0.9838973
B     36       0x4400020000004001  1258594181.028052  1258594183.937122  2.909070  1           0                 20:b7:0d:5d:25:d7;c9:85:3a:55:c3:52                    40:36:58:4a:06:7c;41:9b:2c:e9:b8:c9                    0x0800           143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        476         221          666282       5408          1282      1400      1399.752    5.360048    0       0.079432  0.006111494  0.02068904  163.6262  229036.1  0.3658537   0.9838973
A     36       0x4400020000004000  1258594184.015600  1258594185.427968  1.412368  1           0                 bb:7b:2c:4e:c5:26;3f:ff:ed:4c:fe:48;ac:50:de:51:ac:5e  54:51:0a:5d:0c:a1;fc:c7:f5:8a:e0:1d;a4:b4:6e:7c:d4:ef  0x0800           192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        91          155          2528         214765        20        48        27.78022    9.071207    0       0.079954  0.01552053   0.03033226  64.43079  1789.902  -0.2601626  -0.9767319
B     36       0x4400000000004001  1258594184.015154  1258594185.427506  1.412352  1           0                 20:b7:0d:5d:25:d7;25:06:08:2b:06:01;2c:2d:49:d0:30:82  40:36:58:4a:06:7c;01:04:29:30:27:30;00:00:00:00:00:00  0x0800           143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        155         91           214765       2528          20        1400      1385.581    129.1754    0       0.079416  0.009111947  0.02447955  109.746   152062    0.2601626   0.9767319
A     36       0x4400000000004000  1258594191.015208  1258594191.015208  0.000000  1           5                 70:6c:65:74:65:2e                                      65:72:20:63:6f:6d                                      0x0800           192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        1           0            20           0             20        20        20          0           0       0         0            0           0         0         1           1

Conclusion

Do not forget to reset the config to default for the other tutorials.

t2conf --reset -a && t2build -R

Have fun experimenting!