PCAP extraction and upscaling

pcap

Introduction

This tutorial describes the reduction of pcaps to the very significant packets to answer a specific question. Did it happen to you that your pcap was in the TByte range and you had no clue what’s in it and loading it into Wireshark is already at 1GB cumbersome. Or did you have a pcap recorded with 10GBit/s and you need a pcap with the same content and packet/flow statistics with 10 or even 100GBit/s? Then you are right here.

So one task is to reduce the pcap to the very significant part, hence downsize it to a manageable size. This is what we do everyday, so I had to find a way to solve that problem. There a two options:

pcapd is older and extracts packets into a new pcap according to flow indices in different operational modes of the Anteater. E.g. if T2 is in alarm mode pcaps are only extracted if an alarm in an internal signalling block globalWarn is set. It was designed for maximum flexibility to enable the user to configure T2 into an intelligent flow based IDS. So mostly running on an interface. pcapd works on pcap and pcap-ng. Since version 0.9.0lmw2 it is a factor 100 faster. Forensic guys might have several pcaps, and always different questions, then pcapd has to be invoked every time different flows have to be extracted. This is avoided by findexer. This plugin indexes all packets in the pcap. Currently it can only process pcaps. Whenever your question changes you can select flows and store them into a pcap without running T2 again, hence the drill down process is much faster.

The other task of pcap generation can only be achieved by pcapd. We come back to that topic later.

Preparation

First, restore T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins:

t2build -e -y

Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? yes
Plugin folder emptied

Then compile the following plugins:

t2build tranalyzer2 basicFlow basicStats tcpFlags tcpStates protoStats txtSink

...
BUILD SUCCESSFUL

If you did not create a separate data and results directory yet, please do it now in another bash window, that facilitates your workflow:

mkdir ~/data ~/results

The sample PCAPs used in this tutorial can be downloaded here: faf-exercise.pcap.

Please save it in your ~/data folder.

Note that pcapd and findexer were made for reduction of really large pcaps, but we cannot supply here TBytes for you. So imagine the sample pcap is very, very big. Or just use your own if you need something larger.

In addition, sniffing directly from the interface will be required, so have your sudo password ready.

Now you are all set for pcap extraction experiments.

How to select relevant flows for pcap extraction

In order to find interesting flows it is always good practice to look first at the end report and protocol statistics. The rest of the standard plugins also provide pertinent information in the end report (if PLUGIN_REPORT in tranalyzer.h is set to 1, which it is by default).

t2 -r ~/data/faf-exercise.pcap -w ~/results/

Only IPv4 TCP flows from 2009. All symmetric, all bandwidth measure the same, so we have all the packets. Good. Not much traffic, average packet load indicates downloads, and who is responsible? Probably the biggest talker from USA. Let’s look into the Anomaly flags.

tawk -V tcpAnomaly=0x02cc -V tcpFStat=0x4ff1

The tcpAnomaly column with value 0x02cc is to be interpreted as follows:

   bit | tcpAnomaly | Description
   =============================================================================
     2 | 0x0004     | SEQ Fast retransmission
     3 | 0x0008     | Duplicate ACK
     6 | 0x0040     | Sequence number out-of-order
     7 | 0x0080     | Sequence mess, rather spurious Retransmission
     9 | 0x0200     | Previous packet not captured


The tcpFStat column with value 0x4ff1 is to be interpreted as follows:

   bit | tcpFStat | Description
   =============================================================================
     0 | 0x0001   | Packet good for inter-distance assessment
     4 | 0x0010   | Window state-machine initialized
     5 | 0x0020   | Window update
     6 | 0x0040   | Win 0 probe
     7 | 0x0080   | Win 0 probe ACK
     8 | 0x0100   | Min Window detected
     9 | 0x0200   | WS used
    10 | 0x0400   | Window full
    11 | 0x0800   | Window state-machine count up(1)/down(0)
    14 | 0x4000   | TCP Selective ACK Option

There are definitely problems with flow control, some retries, and the window size hit 0 for 3 times, nevertheless not a security problem.

Looking into the *_protocols.txt* file from the protoStats plugin reveals three well known ports: FTP, SMTP and HTTP.

cat ~/results/faf-exercise_protocols.txt

# Total packets: 5902 (5.90 K)
# Total bytes: 4993414 (4.99 M)
# L2/3 Protocol                            Packets                                   Bytes          Description
0x0800                              5902 [100.00%]                       4993414 [100.00%]          Internet Protocol version 4 (IPv4)


# Total IPv4 packets: 5902 (5.90 K) [100.00%]
# Total IPv6 packets: 0 [0.00%]
# L4 Protocol                              Packets                                   Bytes          Description
  6                                 5902 [100.00%]                       4993414 [100.00%]          Transmission Control Protocol (TCP)


# Total TCP packets: 5902 (5.90 K) [100.00%]
# Total TCP bytes: 4993414 (4.99 M) [100.00%]
# TCP Port                                 Packets                                   Bytes          Description
   21                                 22 [  0.37%]                          2595 [  0.05%]          File Transfer [Control]
   25                                894 [ 15.15%]                        148980 [  2.98%]          Simple Mail Transfer Protocol (SMTP)
   80                                371 [  6.29%]                        304381 [  6.10%]          World Wide Web HTTP
64334                               4615 [ 78.19%]                       4537458 [ 90.87%]


# Total UDP packets: 0 [0.00%]
# Total UDP bytes: 0 [0.00%]

Unencrypted protocols, not uncommon in 2009. Let’s assume FTP and HTTP is unusual for our network, we lately transferred to SFTP and HTTPS internally.

So we would like to load the plugins ftpDecode and httpSniffer. The mail we ignore for now, homework for the reader.

t2build ftpDecode httpSniffer

And execute t2 with that configuration:

t2 -r ~/data/faf-exercise.pcap -w ~/results/

...
--------------------------------------------------------------------------------
basicStats: Biggest L3 talker: 143.166.11.10 (US): 3101 (3.10 K) [52.54%] packets
basicStats: Biggest L3 talker: 143.166.11.10 (US): 4436320 (4.44 M) [88.84%] bytes
tcpFlags: Aggregated ipFlags=0x0044
tcpFlags: Aggregated tcpFStat=0x4ff1
tcpFlags: Aggregated tcpFlags=0x071f
tcpFlags: Aggregated tcpAnomaly=0x02cc
tcpFlags: Number of TCP SYN retries, seq retries: 0, 27
tcpFlags: Number WinSz below 1: 3 [0.05%]
tcpStates: Aggregated tcpStatesAFlags=0x4a
ftpDecode: Aggregated ftpStat=0x0b
ftpDecode: Number of FTP control packets: 22 [0.37%]
ftpDecode: Number of FTP-DATA packets: 4612 (4.61 K) [78.14%]
smtpDecode: Aggregated smtpStat=0x01
smtpDecode: Number of SMTP packets: 894 [15.15%]
smtpDecode: Number of SMTP files: 3
httpSniffer: Aggregated httpStat=0x002c
httpSniffer: Aggregated httpAFlags=0x5000
httpSniffer: Aggregated httpCFlags=0x0010
httpSniffer: Aggregated httpHeadMimes=0x0045
httpSniffer: Number of files img_vid_aud_msg_txt_app_unk: 3_0_0_0_10_7_0
httpSniffer: Number of HTTP packets: 3821 (3.82 K) [64.74%]
httpSniffer: Number of HTTP GET  requests: 16 [0.42%]
--------------------------------------------------------------------------------
...

If you want to decode the hex flags, use tawk -V option. If you do not want to write the full name of the variable, just start with, e.g., tcp and hit tab two times to trigger the autocompletion:

tawk -V ftpStat=0x0b

The ftpStat column with value 0x0b is to be interpreted as follows:

   bit | ftpStat | Description
   =============================================================================
     0 | 0x01    | FTP control port found
     1 | 0x02    | FTP passive parent flow
     3 | 0x08    | FTP active parent flow

Not abnormal. And the HTTP anomaly flags (httpAFlags):

tawk -V httpAFlags=0x5000

The httpAFlags column with value 0x5000 is to be interpreted as follows:

   bit | httpAFlags | Description
   =============================================================================
    12 | 0x1000     | Possible EXE download
    14 | 0x4000     | HTTP 1.0

So there is HTTP/1.0, very odd for 2009, and we download an executable. Not good!

What type of HTTP content are we facing here?

tawk -V httpHeadMimes=0x0045

The httpHeadMimes column with value 0x0045 is to be interpreted as follows:

   bit | httpHeadMimes | Description
   =============================================================================
     0 | 0x0001        | Application
     2 | 0x0004        | Image
     6 | 0x0040        | Text

So applications, images and text. Let’s look at some pictures. Security guys should be interested now. And we have passive FTP around. According to company policy we should only see SSH and SFTP. So let’s dig into the flows.

So this is our question in tawk to the flow file:

tawk '{ if (bitsanyset($ftpStat, 0x0f) || bitsanyset($httpAFlags, 0x5000)) { print } else { split($httpImg_Vid_Aud_Msg_Txt_App_Unk, A, "_"); if (A[1] > 0) print } }’ ~/results/faf-exercise_flows.txt | tcol

%dir  flowInd  flowStat            timeFirst          timeLast           duration    numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP           srcIPCC  srcIPOrg                       srcPort  dstIP           dstIPCC  dstIPOrg                       dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT       stdIAT       pktps       bytps     pktAsm       bytAsm      tcpFStat  ipMindIPID  ipMaxdIPID  ipMinTTL  ipMaxTTL  ipTTLChg  ipToS  ipFlags  ipOptCnt  ipOptCpCl_Num    ip6OptCntHH_D  ip6OptHH_D             tcpISeqN    tcpPSeqCnt  tcpSeqSntBytes  tcpSeqFaultCnt  tcpPAckCnt  tcpFlwLssAckRcvdBytes  tcpAckFaultCnt  tcpBFlgtMx  tcpInitWinSz  tcpAveWinSz  tcpMinWinSz  tcpMaxWinSz  tcpWinSzDwnCnt  tcpWinSzUpCnt  tcpWinSzChgDirCnt  tcpWinSzThRt  tcpFlags  tcpAnomaly  tcpOptPktCnt  tcpOptCnt  tcpOptions  tcpMSS  tcpWS  tcpMPTBF  tcpMPF  tcpMPAID  tcpMPdssF  tcpTmS  tcpTmER  tcpEcI  tcpUtm    tcpBtm    tcpSSASAATrip  tcpRTTAckTripMin  tcpRTTAckTripMax  tcpRTTAckTripAve  tcpRTTAckTripJitAve  tcpRTTSseqAA  tcpRTTAckJitAve  tcpStatesAFlags  ftpStat  ftpCDFindex  ftpCC                          ftpRC                            ftpNumUser  ftpUser      ftpNumPass  ftpPass    ftpNumCP  ftpCP                                                 httpStat  httpAFlags  httpMethods  httpHeadMimes  httpCFlags  httpGet_Post  httpRSCnt  httpRSCode  httpURL_Via_Loc_Srv_Pwr_UAg_XFr_Ref_Cky_Mim  httpImg_Vid_Aud_Msg_Txt_App_Unk  httpHosts     httpURL                                   httpMimes                   httpCookies  httpImages                                                                              httpVideos  httpAudios  httpMsgs  httpAppl                                                                                           httpText                                                                               httpPunk  httpBdyURL  httpUsrAg                                                                                                             httpXFor  httpRefrr  httpVia        httpLoc                                                              httpServ                 httpPwr
B     3        0x0400000000004001  1258544216.915576  1258544217.008019  0.092443    1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              198.189.255.75  us       "California State University"  80       192.168.1.104   07       "Private network"              1260     6        73          18           95603        319           0         1380      1309.63     274.7284    0       0.021251  0.001266342  0.003059902  789.6758    1034183   0.6043956    0.9933488   0x0811    1           42177       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  4272559795  73          95603           0               73          319                    0               16560       5840          6432         5840         6432         0               1              1                  0             0x031b    0x0000      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.00045        7.8e-05           0.014188          0.001192556       0.002669854          0.00849205    0.002675539      0x00             0x00                                                                                  0                        0                      0                                                               0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                            "application/octet-stream"                                                                                                                                         "_softw_90_update_u7avi1777u1705ff.bin_3_B_1_0"                                                                                                                                                                                                                                                                                                                                                                                                "Apache"
B     5        0x0400000000004001  1258544217.357036  1258544217.413505  0.056469    1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              198.189.255.75  us       "California State University"  80       192.168.1.104   07       "Private network"              1262     6        26          9            30820        320           0         1380      1185.385    425.1664    0       0.017243  0.002171885  0.00369694   460.4296    545786.2  0.4857143    0.9794477   0x0811    1           20242       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  4278713148  26          30820           0               26          320                    0               9660        5840          6431.921     5840         6432         0               1              1                  0             0x031b    0x0000      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000462       0.000214          0.010205          0.001545667       0.002568446          0.008867128   0.002571311      0x00             0x00                                                                                  0                        0                      0                                                               0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                            "application/octet-stream"                                                                                                                                         "_softw_90_update_u7iavi2511u2510ff.bin_5_B_1_0"                                                                                                                                                                                                                                                                                                                                                                                               "Apache"
B     7        0x0400000000004001  1258544217.763049  1258544217.791016  0.027967    1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              198.189.255.75  us       "California State University"  80       192.168.1.104   07       "Private network"              1264     6        7           6            5268         317           0         1380      752.5714    567.7107    0       0.017745  0.003995285  0.005150571  250.295     188364.9  0.07692308   0.8864816   0x0811    1           53026       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  4269177574  7           5268            0               7           317                    0               5268        5840          6362.352     5840         6432         0               1              1                  0             0x031b    0x0000      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000445       0.000423          0.010686          0.0025205         0.003097677          0.009895643   0.003103809      0x00             0x00                                                                                  0                        0                      0                                                               0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                            "application/octet-stream"                                                                                                                                         "_softw_90_update_x8xplsb2_118c8.bin_7_B_1_0"                                                                                                                                                                                                                                                                                                                                                                                                  "Apache"
B     9        0x0400000000004001  1258544218.137543  1258544218.165782  0.028239    1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              198.189.255.75  us       "California State University"  80       192.168.1.104   07       "Private network"              1266     6        5           6            1690         320           0         1380      338         441.2029    0       0.017993  0.005647799  0.005497161  177.0601    59846.31  -0.09090909  0.681592    0x0811    1           30104       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  4270113443  5           1690            0               5           320                    0               1690        5840          6289.861     5840         6432         0               1              1                  0             0x031b    0x0000      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000466       0.000456          0.010956          0.002583333       0.003173773          0.009871334   0.003179277      0x00             0x00                                                                                  0                        0                      0                                                               0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                            "application/octet-stream"                                                                                                                                         "_softw_90_update_x8xplsc_149d148c8.bin_9_B_1_0"                                                                                                                                                                                                                                                                                                                                                                                               "Apache"
B     11       0x0400000000004001  1258562478.266384  1258562509.653978  31.387594   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11   us       "Mozilla Corporation"          80       192.168.1.104   07       "Private network"              1384     6        8           7            4184         476           0         1380      523         579.3142    0       23.04     3.923449     7.138864     0.2548778   133.3011  0.06666667   0.7957082   0x0811    59415       6116        52        52        0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  531087527   8           4184            0               8           476                    0               2760        5840          6383.246     5840         6432         0               1              1                  0             0x031b    0x0000      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000411       0.000406          8.196325          1.171333          2.655198             4.101608      7.522519         0x00             0x00                                                                                  0                        0                      0                                                               0x0068    0x0000      0x00         0x0004         0x0010      0_0           1          200         0_1_0_1_0_0_0_0_0_1                          1_0_0_0_0_0_0                                                                            "image/png"                              "_en-US_thunderbird_2.0.0.23_start_thunderbird_startpage.png_11_B_1_0"                                                                                                                                                                                                                                                                                                                                                                                                               "1.1 varnish"                                                                       "Apache/2.2.3 (CentOS)"
B     10       0x0400000000004001  1258562467.754689  1258562509.653962  41.899273   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11   us       "Mozilla Corporation"          80       192.168.1.104   07       "Private network"              1379     6        18          14           15606        1801          0         1380      867         568.1608    0       22.97829  2.327737     5.515566     0.4296017   372.4647  0.125        0.7930717   0x0811    9916        55605       52        52        0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3267889532  18          15606           0               18          1801                   0               4140        5840          9616.309     5840         9648         0               4              1                  0             0x031b    0x0000      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000207       0.000186          9.952202          1.338241          2.93412              2.651285      5.881808         0x00             0x00                                                                                  0                        0                      0                                                               0x0068    0x0000      0x00         0x0044         0x0010      0_0           2          302;200     0_1_1_1_1_0_0_0_0_2                          2_0_0_0_2_0_0                                                                            "text/html";"image/jpeg"                 "_style_dalvay_bg-header-small.jpg_10_B_4_0";"_style_dalvay_main-feature.jpg_10_B_5_1"                                                                                                                                       "_thunderbird_2.0.0.23_start__10_B_1_0";"_en-US_thunderbird_2.0.0.23_start__10_B_2_1"                                                                                                                                                                   "1.1 varnish"  "http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/"  "Apache/2.2.3 (CentOS)"  "PHP/5.1.6"
A     33       0x0400000000004000  1258587444.865917  1258587445.631435  0.765518    1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104   07       "Private network"              1908     198.189.255.75  us       "California State University"  80       6        24          74           319          97078         0         319       13.29167    53.98931    0       0.574143  0.03189658   0.1023528    31.35132    416.7113  -0.5102041   -0.9934495  0x0011    1           2           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3626872219  23          319             0               23          95699                  1               319         65535         65535        65535        65535        0               0              0                  0             0x011b    0x0008      1             4          0x00000016  1460    1      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.007304       0.007047          0.573488          0.01674978        0.06118462           0.007744      0                0x02             0x00                                                                                  0                        0                      0                                                               0x006c    0x4000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"  "/softw/90/update/u7avi1778u1705z7.bin"                                                                                                                                                                                                                                                                                                                                                                                      "AVGINET9-WXPPX86 90 AVI=270.14.72/2511 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B     33       0x0400000000004001  1258587444.873221  1258587445.638482  0.765261    1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              198.189.255.75  us       "California State University"  80       192.168.1.104   07       "Private network"              1908     6        74          24           97078        319           0         1380      1311.865    268.8814    0       0.573915  0.01034136   0.06162005   96.69904    126856.1  0.5102041    0.9934495   0x0811    33497       31968       57        57        0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  2668953963  72          99839           0               74          319                    0               8280        5840          6432         5840         6432         0               1              1                  0             0x031b    0x00c0      1             4          0x00000016  1380    1      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.00044        0.000183          0.010698          0.0008634167      0.001742813          0.01761319    0.06120944       0x02             0x00                                                                                  0                        0                      0                                                               0x0068    0x5000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_2_0                                                                            "application/octet-stream"                                                                                                                                         "_softw_90_update_u7avi1778u1705z7.bin_33_B_1_0";"_softw_90_update_u7avi1778u1705z7.bin_33_B_5_1"                                                                                                                                                                                                                                                                                                                                              "Apache"
A     34       0x0400000000004000  1258587445.990733  1258587446.040428  0.049695    1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104   07       "Private network"              1910     198.189.255.75  us       "California State University"  80       6        8           19           320          21634         0         320       40          88.41758    0       0.00999   0.006211875  0.002938672  160.982     6439.28   -0.4074074   -0.9708481  0x0011    1           1           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3544152804  7           320             0               7           21635                  0               320         65535         65535        65535        65535        0               0              0                  0             0x011b    0x0000      1             4          0x00000016  1460    1      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.007517       0.007043          0.007573          0.007338999       0.0001176775         0.007978      0                0x02             0x00                                                                                  0                        0                      0                                                               0x006c    0x4000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"  "/softw/90/update/u7iavi2512u2511z7.bin"                                                                                                                                                                                                                                                                                                                                                                                     "AVGINET9-WXPPX86 90 AVI=270.14.72/2511 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B     34       0x0400000000004001  1258587445.998250  1258587446.047471  0.049221    1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              198.189.255.75  us       "California State University"  80       192.168.1.104   07       "Private network"              1910     6        19          8            21634        320           0         1380      1138.632    455.4652    0       0.017522  0.002590579  0.003994345  386.0141    439527.8  0.4074074    0.9708481   0x0811    40938       24582       57        57        0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  2671895030  19          21635           0               19          320                    0               5520        5840          6431.036     5840         6432         0               1              1                  0             0x031b    0x0000      1             4          0x00000016  1380    1      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000461       0.000402          0.010451          0.00174075        0.00275091           0.00907975    0.002753426      0x02             0x00                                                                                  0                        0                      0                                                               0x0068    0x5000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                            "application/octet-stream"                                                                                                                                         "_softw_90_update_u7iavi2512u2511z7.bin_34_B_1_0"                                                                                                                                                                                                                                                                                                                                                                                              "Apache"
A     36       0x0400000000004000  1258594163.408285  1258594191.015208  27.606923   1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105   07       "Private network"              49330    143.166.11.10   us       "Dell"                         64334    6        1514        3101         0            4268858       0         0         0           0           0       5.58724   0.01823444   0.1478493    54.84132    0         -0.3438787   -1          0x49b1    1           223         128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3392384642  1511        0               0               1511        4255056                437             0           8192          45402        0            64860        253             194            254                0.001981506   0x0416    0x0008      511           1536       0x0000003e  1460    4      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.078742       1e-06             0.67088           0.03865783        0.04056465           0.07920299    0                0x42             0x02     35                                                                           0                        0                      1         "/video/R79733.EXE"                                   0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0
B     36       0x0400000000004001  1258594163.487027  1258594185.427506  21.940479   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10   us       "Dell"                         64334    192.168.1.105   07       "Private network"              49330    6        3101        1514         4268858      0             0         1380      1376.607    60.23097    0       0.67109   0.007075286  0.02745561   141.3369    194565.4  0.3438787    1           0x0c51    1           6365        111       111       0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3177226489  2866        5233476         27              3099        0                      0               28980       8192          64860        8192         64860        0               1              1                  0             0x031b    0x02c4      1             4          0x00000016  1380    1      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000461       2e-06             5.587702          0.004777641       0.1447472            0.04343547    0.1503238        0x02             0x02     35                                                                           0                        0                      1         "/video/R79733.EXE"                                   0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0
A     35       0x0400000000004000  1258594162.928342  1258594185.618346  22.690004   1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105   07       "Private network"              49329    143.166.11.10   us       "Dell"                         21       6        11          11           92           1231          0         24        8.363636    8.41835     0       21.78007  2.062728     5.945361     0.484795    4.054649  0            -0.8609222  0x0811    1           2328        128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  2427598871  10          92              0               10          1231                   0               24          8192          62176.56     8192         64860        8               1              9                  0             0x001a    0x0000      1             4          0x00000016  1460    1      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.080252       0.077494          306.0649          29.85102          83.48595             0.08095       0                0x02             0x09     36           USER;PASS;TYPE;PASV;SIZE;RETR                                   1           "anonymous"  1           "IEUser@"  2         "I";"/video/R79733.EXE"                               0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0
B     35       0x0400000000004001  1258594163.008594  1258594491.683288  328.674694  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10   us       "Dell"                         21       192.168.1.105   07       "Private network"              49329    6        11          11           1231         92            0         950       111.9091    232.9224    0       306.2558  29.87952     83.53862     0.03346774  3.745345  0            0.8609222   0x0811    2732        26560       239       239       0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  365320932   11          1231            0               11          92                     0               971         4140          4214.603     4140         4232         0               6              1                  0             0x061e    0x0000      1             2          0x00000014  1380    1      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000698       0.00045           0.194089          0.04303964        0.07018124           29.89405      83.48597         0x42             0x09     36                                          220;331;230;200;227;213;125;226  0                        0                      1         "125 Data connection already open; Transfer startin"  0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0

Clearly unencrypted. And look at flow 35 and 36 an executable R79733.EXE downloaded. Damn bad. Imagine your pcap is 20TB large, then the flow files become large as well, in the GB range.

So you might be interested in reducing the pcap size first to the very interesting protocols. And then go into more detail.

This is what we do in the next section.

Extract flows with pcapd plugin

Extracting flow based information from the pcap to produce a downsized pcap can be achieved by the pcapd plugin. It functions in various modes, which can be configured in pcap.h.

pcapd

vi src/pcapd.h

...
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */

#define PD_MODE_IN  0 // 0: extract flows listed in input file (if -e option was used),
                      //    or extract flows if alarm bit is set (if -e option was not used)
                      // 1: dump all packets

#define PD_EQ       1 // whether to dump matching (1) or non-matching (0) flows
#define PD_OPP      0 // extract also the opposite flow, 0: don't

#define PD_MODE_OUT 0 // 0: one pcap,
                      // 1: one pcap per flow

#define PD_SPLIT    1 // whether or not to split output file (-W option)

#define PD_FORMAT   0 // Format of the input file (-e option):
                      //   0: flow index only,
                      //   1: flow file format

#define PD_TSHFT    0   // Timeshift
#define PDTTSFTS    0   // add by sec
#define PDTTSFTNMS  1   // add usec or nsec, depends on TSTAMP_PREC

#define PD_MACSHFT  0   // Mac shift last byte
#define PDMACSSHFT  1   // add to src mac
#define PDMACDSHFT  1      // add to dst mac

#define PD_VLNSHFT  0   // vlan shift
#define PDVLNSHFT   1   // add value inner vlan

#define PD_IPSHFT   0   // add to src, dst IP
#define PDIP4SHFTN  0x01000000 //  add IP4 32 bit network order
#define PDIP6SHFTN  0x0100000000000000 // add IP6 shift last 64 bit network order

/* +++++++++++++++++++++ ENV / RUNTIME - conf Variables +++++++++++++++++++++ */

#define PD_MAX_FD 128           // Maximum number of simultaneously open file descriptors
#define PD_SUFFIX "_pcapd.pcap" // extension for generated pcap file

/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...

The constant PD_SPLIT controls the size of the resulting pcap files if the -W option is selected in the T2 command line. Not important here, so leave it at default setting.

PD_MODE_IN controls the basic mode of operation. The idea behind the first mode (0) and the -e option is to use tawk to extract flows of interest and then the pcapd plugin to create a PCAP with all those flows.

The first column must be the flow index, the rest is ignored Lines starting with %, #, a space or a tab are ignored, along with empty lines.

Flows whose index appears in the -e file will be written to a file named PREFIX_PD_SUFFIX, where PREFIX is the value given to T2 in the -e option.

Note that if PD_EQ is set to 0, the flows whose index does NOT appear in the file will be dumped.

If the FL_ALARM bit is set by any plugin, such as regex_pcre or dnsDecode, PD_OPP defines whether also the opposite flow is extracted, where the very alarm did not occur. This is important for forensics and IDS mode purposes if you want to see the full history of an alarm. It is also active when reading a flow index file, so we will switch it on.

If PD_MODE_IN is set to 1 and the -e option is NOT provided, a special alarm mode is activated where any plugin can trigger pcapd to dump a flow. Will be discussed in the alarm mode tutorial.

PD_MAX_FD controls the number of active file handles, so that the number of open files does not exceed the OS limits. This comes in handy when PD_MODE_OUT is set to 1. Then all packets of each flow are stored in a separate file.

We leave everything at default, because we work now on a pcap and generated a flow index file: ~/results/nudel. Hence, we need the -e option in the T2 command line later on.

In order to accelerate the extraction process it is recommended to build pcapd under another directory, so that only this plugin can be loaded. Hence, if another question has to be applied to the pcap only the plugin folder needs be selected by the -p option. Because pcapd is only needed temporarily, it can be built under /tmp. Sure, you can build it with the other plugins, and it would be easier, but then you execute all the other plugins loaded… a waste of time!

So for the sake of the exercise I’ll show you how different configurations are handled in parallel. Hence, we build an extra pcapd and configure it to extract both parts of a flow.

t2conf pcapd -D PD_OPP=1

t2build -p /tmp tranalyzer2 pcapd

Move to your results directory window now and produce a file which contains all interesting flow indexes selected earlier by tawk. Just add the variable $flowInd in the print statement as shown below:

tawk -H '{ if (bitsanyset($ftpStat, 0x0f) || bitsanyset($httpAFlags, 0x5000)) { print $flowInd } else { split($httpImg_Vid_Aud_Msg_Txt_App_Unk, A, "_"); if (A[1] > 0) print $flowInd } }' ~/results/faf-exercise_flows.txt | uniq > ~/results/nudel

cat ~/results/nudel

3
5
7
9
11
10
33
34
36
35

And tell pcapd that it should load the flow index file from the results directory with the -e option:

t2 -r ~/data/faf-exercise.pcap -e ~/results/nudel -p /tmp

================================================================================
Tranalyzer 0.9.0(Anteater), Tarantula. PID: 6853
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: pcapd, 0.9.0
[INF] IPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 406105 (406.11 K)
[INF] IPv6 Ver: 5, Rev: 17122020, Range Mode: 0, subnet ranges loaded: 51345 (51.34 K)
[INF] pcapd: 10 flow indices
...
--------------------------------------------------------------------------------
pcapd: number of packets extracted: 4958 (4.96 K) [84.01%]
--------------------------------------------------------------------------------
...

As you can see T2 produces a new pcap called ~/results/nudel_pcapd.pcap. Rerun T2 on the new pcap and now we can also produce a packet file, because it now contains only flows of interest.

ls ~/results

faf-exercise_flows.txt  faf-exercise_headers.txt  faf-exercise_packets.txt  faf-exercise_protocols.txt
nudel  nudel_pcapd.pcap

t2 -r ~/results/nudel_pcapd.pcap -w ~/results/nudel -s

20 flows instead of 72. If you look at the resulting flow file, you actually see more flows than in our tawk request on the original flow file above. Because not in every flow direction certain flags are set. So you also get the return or originating flow now, which preserves context of a communication and avoids unnecessary further tawk questions.

tcol ~/results/nudel_flows.txt

%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP           srcIPCC  srcIPOrg                       srcPort  dstIP           dstIPCC  dstIPOrg                       dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT       stdIAT       pktps      bytps     pktAsm       bytAsm      tcpFStat  ipMindIPID  ipMaxdIPID  ipMinTTL  ipMaxTTL  ipTTLChg  ipToS  ipFlags  ipOptCnt  ipOptCpCl_Num    ip6OptCntHH_D  ip6OptHH_D             tcpISeqN    tcpPSeqCnt  tcpSeqSntBytes  tcpSeqFaultCnt  tcpPAckCnt  tcpFlwLssAckRcvdBytes  tcpAckFaultCnt  tcpBFlgtMx  tcpInitWinSz  tcpAveWinSz  tcpMinWinSz  tcpMaxWinSz  tcpWinSzDwnCnt  tcpWinSzUpCnt  tcpWinSzChgDirCnt  tcpWinSzThRt  tcpFlags  tcpAnomaly  tcpOptPktCnt  tcpOptCnt  tcpOptions  tcpMSS  tcpWS  tcpMPTBF  tcpMPF  tcpMPAID  tcpMPdssF  tcpTmS  tcpTmER  tcpEcI  tcpUtm    tcpBtm    tcpSSASAATrip  tcpRTTAckTripMin  tcpRTTAckTripMax  tcpRTTAckTripAve  tcpRTTAckTripJitAve  tcpRTTSseqAA  tcpRTTAckJitAve  tcpStatesAFlags  ftpStat  ftpCDFindex  ftpCC                          ftpRC                            ftpNumUser  ftpUser      ftpNumPass  ftpPass    ftpNumCP  ftpCP                                                 httpStat  httpAFlags  httpMethods  httpHeadMimes  httpCFlags  httpGet_Post  httpRSCnt  httpRSCode  httpURL_Via_Loc_Srv_Pwr_UAg_XFr_Ref_Cky_Mim  httpImg_Vid_Aud_Msg_Txt_App_Unk  httpHosts                   httpURL                                                                                                                                   httpMimes                   httpCookies  httpImages                                                                            httpVideos  httpAudios  httpMsgs  httpAppl                                                                                         httpText                                                                             httpPunk  httpBdyURL  httpUsrAg                                                                                                             httpXFor  httpRefrr                                                            httpVia        httpLoc                                                              httpServ                 httpPwr
A     1        0x0400000000004000  1258544216.908284  1258544217.008468  0.100184   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104   07       "Private network"              1260     198.189.255.75  us       "California State University"  80       6        18          73           319          95603         0         319       17.72222    61.73868    0       0.013738  0.005565778  0.00363383   179.6694   3184.141  -0.6043956   -0.9933488  0x0011    1           2           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  1942699277  17          319             0               17          95603                  0               319         65535         65535        65535        65535        0               0              0                  0             0x011b    0x0000      1             4          0x00000016  1460    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.007292       0.006796          0.007834          0.007299494       0.0001743189         0.007742      0                0x00             0x00                                                                                  0                        0                      0                                                               0x006c    0x0000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"                "/softw/90/update/u7avi1777u1705ff.bin"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                "AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B     1        0x0400000000004001  1258544216.915576  1258544217.008019  0.092443   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              198.189.255.75  us       "California State University"  80       192.168.1.104   07       "Private network"              1260     6        73          18           95603        319           0         1380      1309.63     274.7284    0       0.021251  0.001266342  0.003059902  789.6758   1034183   0.6043956    0.9933488   0x0811    1           42177       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  4272559795  73          95603           0               73          319                    0               16560       5840          6432         5840         6432         0               1              1                  0             0x031b    0x0000      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.00045        7.8e-05           0.014188          0.001192556       0.002669854          0.00849205    0.002675539      0x00             0x00                                                                                  0                        0                      0                                                               0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                                                                                                                                          "application/octet-stream"                                                                                                                                       "_softw_90_update_u7avi1777u1705ff.bin_1_B_1_0"                                                                                                                                                                                                                                                                                                                                                                                                                                                      "Apache"
A     2        0x0400000000004000  1258544217.349751  1258544217.413719  0.063968   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104   07       "Private network"              1262     198.189.255.75  us       "California State University"  80       6        9           26           320          30820         0         320       35.55556    84.1992     0       0.009743  0.007107555  0.002012674  140.6953   5002.501  -0.4857143   -0.9794477  0x0011    1           2           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  998473205   8           320             0               8           30820                  0               320         65535         65300.4      65075        65535        1               0              0                  0             0x011b    0x0000      1             4          0x00000016  1460    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.007285       0.007036          0.00757           0.007321462       0.0001213493         0.007747      0                0x00             0x00                                                                                  0                        0                      0                                                               0x006c    0x0000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"                "/softw/90/update/u7iavi2511u2510ff.bin"                                                                                                                                                                                                                                                                                                                                                                                                                                                                               "AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B     2        0x0400000000004001  1258544217.357036  1258544217.413505  0.056469   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              198.189.255.75  us       "California State University"  80       192.168.1.104   07       "Private network"              1262     6        26          9            30820        320           0         1380      1185.385    425.1664    0       0.017243  0.002171885  0.00369694   460.4296   545786.2  0.4857143    0.9794477   0x0811    1           20242       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  4278713148  26          30820           0               26          320                    0               9660        5840          6431.921     5840         6432         0               1              1                  0             0x031b    0x0000      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000462       0.000214          0.010205          0.001545667       0.002568446          0.008867128   0.002571311      0x00             0x00                                                                                  0                        0                      0                                                               0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                                                                                                                                          "application/octet-stream"                                                                                                                                       "_softw_90_update_u7iavi2511u2510ff.bin_2_B_1_0"                                                                                                                                                                                                                                                                                                                                                                                                                                                     "Apache"
A     3        0x0400000000004000  1258544217.755746  1258544217.791475  0.035729   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104   07       "Private network"              1264     198.189.255.75  us       "California State University"  80       6        6           7            317          5268          0         317       52.83333    98.10986    0       0.010241  0.005954833  0.002865114  167.9308   8872.345  -0.07692308  -0.8864816  0x0011    1           2           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  935235267   5           317             0               5           5268                   0               317         65535         65535        65535        65535        0               0              0                  0             0x011b    0x0000      1             4          0x00000016  1460    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.007303       0.007039          0.007577          0.007375143       0.0001950087         0.007748      0                0x00             0x00                                                                                  0                        0                      0                                                               0x006c    0x0000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"                "/softw/90/update/x8xplsb2_118c8.bin"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  "AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B     3        0x0400000000004001  1258544217.763049  1258544217.791016  0.027967   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              198.189.255.75  us       "California State University"  80       192.168.1.104   07       "Private network"              1264     6        7           6            5268         317           0         1380      752.5714    567.7107    0       0.017745  0.003995285  0.005150571  250.295    188364.9  0.07692308   0.8864816   0x0811    1           53026       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  4269177574  7           5268            0               7           317                    0               5268        5840          6362.352     5840         6432         0               1              1                  0             0x031b    0x0000      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000445       0.000423          0.010686          0.0025205         0.003097677          0.009895643   0.003103809      0x00             0x00                                                                                  0                        0                      0                                                               0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                                                                                                                                          "application/octet-stream"                                                                                                                                       "_softw_90_update_x8xplsb2_118c8.bin_3_B_1_0"                                                                                                                                                                                                                                                                                                                                                                                                                                                        "Apache"
A     4        0x0400000000004000  1258544218.130258  1258544218.166240  0.035982   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104   07       "Private network"              1266     198.189.255.75  us       "California State University"  80       6        6           5            320          1690          0         320       53.33333    99.03834    0       0.01049   0.005997     0.00290767   166.75     8893.335  0.09090909   -0.681592   0x0011    1           1           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  1909828099  5           320             0               5           1690                   0               320         65535         65535        65535        65535        0               0              0                  0             0x011b    0x0000      1             4          0x00000016  1460    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.007285       0.007037          0.007543          0.007288          0.0001869964         0.007751      0                0x00             0x00                                                                                  0                        0                      0                                                               0x006c    0x0000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"                "/softw/90/update/x8xplsc_149d148c8.bin"                                                                                                                                                                                                                                                                                                                                                                                                                                                                               "AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B     4        0x0400000000004001  1258544218.137543  1258544218.165782  0.028239   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              198.189.255.75  us       "California State University"  80       192.168.1.104   07       "Private network"              1266     6        5           6            1690         320           0         1380      338         441.2029    0       0.017993  0.005647799  0.005497161  177.0601   59846.31  -0.09090909  0.681592    0x0811    1           30104       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  4270113443  5           1690            0               5           320                    0               1690        5840          6289.861     5840         6432         0               1              1                  0             0x031b    0x0000      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000466       0.000456          0.010956          0.002583333       0.003173773          0.009871334   0.003179277      0x00             0x00                                                                                  0                        0                      0                                                               0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                                                                                                                                          "application/octet-stream"                                                                                                                                       "_softw_90_update_x8xplsc_149d148c8.bin_4_B_1_0"                                                                                                                                                                                                                                                                                                                                                                                                                                                     "Apache"
A     6        0x0400000000004000  1258562478.204828  1258562509.633367  31.428539  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104   07       "Private network"              1384     63.245.221.11   us       "Mozilla Corporation"          80       6        7           8            476          4184          0         476       68          138.7919    0       23.04004  4.489791     7.379        0.2227275  15.14547  -0.06666667  -0.7957082  0x0011    1           109         128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  654559040   6           476             0               6           4184                   0               476         65535         65535        65535        65535        0               0              0                  0             0x011b    0x0000      1             4          0x00000016  1460    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.061556       0.020611          23.03958          2.930275          7.038339             0.061967      0                0x00             0x00                                                                                  0                        0                      0                                                               0x006c    0x0000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_1_0_0                          0_0_0_0_0_0_0                    "www.mozillamessaging.com"  "/en-US/thunderbird/2.0.0.23/start/thunderbird_startpage.png"                                                                                                                                                                                                                                                                                                                                                                                                                                                          "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.23) Gecko/20090812 Thunderbird/2.0.0.23"                              "http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/"
B     6        0x0400000000004001  1258562478.266384  1258562509.653978  31.387594  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11   us       "Mozilla Corporation"          80       192.168.1.104   07       "Private network"              1384     6        8           7            4184         476           0         1380      523         579.3142    0       23.04     3.923449     7.138864     0.2548778  133.3011  0.06666667   0.7957082   0x0811    59415       6116        52        52        0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  531087527   8           4184            0               8           476                    0               2760        5840          6383.246     5840         6432         0               1              1                  0             0x031b    0x0000      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000411       0.000406          8.196325          1.171333          2.655198             4.101608      7.522519         0x00             0x00                                                                                  0                        0                      0                                                               0x0068    0x0000      0x00         0x0004         0x0010      0_0           1          200         0_1_0_1_0_0_0_0_0_1                          1_0_0_0_0_0_0                                                                                                                                                                                          "image/png"                              "_en-US_thunderbird_2.0.0.23_start_thunderbird_startpage.png_6_B_1_0"                                                                                                                                                                                                                                                                                                                                                                                                                                                                    "1.1 varnish"                                                                       "Apache/2.2.3 (CentOS)"
A     5        0x0400000000004000  1258562467.749142  1258562509.633370  41.884228  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104   07       "Private network"              1379     63.245.221.11   us       "Mozilla Corporation"          80       6        14          18           1801         15606         0         455       128.6429    183.7502    0       22.97809  2.991731     6.053773     0.3342547  42.99948  -0.125       -0.7930717  0x0811    1           91          128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3115250060  13          1801            0               13          15606                  0               455         65535         65497.7      64737        65535        2               2              2                  0             0x011b    0x0000      1             4          0x00000016  1460    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.005547       7.1e-05           22.97788          1.313044          5.097706             0.005754      0                0x00             0x00                                                                                  0                        0                      0                                                               0x006c    0x0000      0x02         0x0000         0x0010      4_0           0                      4_0_0_0_0_1_0_1_0_0                          0_0_0_0_0_0_0                    "www.mozillamessaging.com"  "/thunderbird/2.0.0.23/start/";"/en-US/thunderbird/2.0.0.23/start/";"/style/dalvay/bg-header-small.jpg";"/style/dalvay/main-feature.jpg"                                                                                                                                                                                                                                                                                                                                                                               "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.23) Gecko/20090812 Thunderbird/2.0.0.23"                              "http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/"
B     5        0x0400000000004001  1258562467.754689  1258562509.653962  41.899273  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11   us       "Mozilla Corporation"          80       192.168.1.104   07       "Private network"              1379     6        18          14           15606        1801          0         1380      867         568.1608    0       22.97829  2.327737     5.515566     0.4296017  372.4647  0.125        0.7930717   0x0811    9916        55605       52        52        0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3267889532  18          15606           0               18          1801                   0               4140        5840          9616.309     5840         9648         0               4              1                  0             0x031b    0x0000      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000207       0.000186          9.952202          1.338241          2.93412              2.651285      5.881808         0x00             0x00                                                                                  0                        0                      0                                                               0x0068    0x0000      0x00         0x0044         0x0010      0_0           2          302;200     0_1_1_1_1_0_0_0_0_2                          2_0_0_0_2_0_0                                                                                                                                                                                          "text/html";"image/jpeg"                 "_style_dalvay_bg-header-small.jpg_5_B_4_0";"_style_dalvay_main-feature.jpg_5_B_5_1"                                                                                                                                     "_thunderbird_2.0.0.23_start__5_B_1_0";"_en-US_thunderbird_2.0.0.23_start__5_B_2_1"                                                                                                                                                                                                                             "1.1 varnish"  "http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/"  "Apache/2.2.3 (CentOS)"  "PHP/5.1.6"
A     7        0x0400000000004000  1258587444.865917  1258587445.631435  0.765518   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104   07       "Private network"              1908     198.189.255.75  us       "California State University"  80       6        24          74           319          97078         0         319       13.29167    53.98931    0       0.574143  0.03189658   0.1023528    31.35132   416.7113  -0.5102041   -0.9934495  0x0011    1           2           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3626872219  23          319             0               23          95699                  1               319         65535         65535        65535        65535        0               0              0                  0             0x011b    0x0008      1             4          0x00000016  1460    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.007304       0.007047          0.573488          0.01674978        0.06118462           0.007744      0                0x02             0x00                                                                                  0                        0                      0                                                               0x006c    0x4000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"                "/softw/90/update/u7avi1778u1705z7.bin"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                "AVGINET9-WXPPX86 90 AVI=270.14.72/2511 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B     7        0x0400000000004001  1258587444.873221  1258587445.638482  0.765261   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              198.189.255.75  us       "California State University"  80       192.168.1.104   07       "Private network"              1908     6        74          24           97078        319           0         1380      1311.865    268.8814    0       0.573915  0.01034136   0.06162005   96.69904   126856.1  0.5102041    0.9934495   0x0811    33497       31968       57        57        0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  2668953963  72          99839           0               74          319                    0               8280        5840          6432         5840         6432         0               1              1                  0             0x031b    0x00c0      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.00044        0.000183          0.010698          0.0008634167      0.001742813          0.01761319    0.06120944       0x02             0x00                                                                                  0                        0                      0                                                               0x0068    0x5000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_2_0                                                                                                                                                                                          "application/octet-stream"                                                                                                                                       "_softw_90_update_u7avi1778u1705z7.bin_7_B_1_0";"_softw_90_update_u7avi1778u1705z7.bin_7_B_5_1"                                                                                                                                                                                                                                                                                                                                                                                                      "Apache"
A     8        0x0400000000004000  1258587445.990733  1258587446.040428  0.049695   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104   07       "Private network"              1910     198.189.255.75  us       "California State University"  80       6        8           19           320          21634         0         320       40          88.41758    0       0.00999   0.006211875  0.002938672  160.982    6439.28   -0.4074074   -0.9708481  0x0011    1           1           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3544152804  7           320             0               7           21635                  0               320         65535         65535        65535        65535        0               0              0                  0             0x011b    0x0000      1             4          0x00000016  1460    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.007517       0.007043          0.007573          0.007338999       0.0001176775         0.007978      0                0x02             0x00                                                                                  0                        0                      0                                                               0x006c    0x4000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"                "/softw/90/update/u7iavi2512u2511z7.bin"                                                                                                                                                                                                                                                                                                                                                                                                                                                                               "AVGINET9-WXPPX86 90 AVI=270.14.72/2511 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B     8        0x0400000000004001  1258587445.998250  1258587446.047471  0.049221   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              198.189.255.75  us       "California State University"  80       192.168.1.104   07       "Private network"              1910     6        19          8            21634        320           0         1380      1138.632    455.4652    0       0.017522  0.002590579  0.003994345  386.0141   439527.8  0.4074074    0.9708481   0x0811    40938       24582       57        57        0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  2671895030  19          21635           0               19          320                    0               5520        5840          6431.036     5840         6432         0               1              1                  0             0x031b    0x0000      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000461       0.000402          0.010451          0.00174075        0.00275091           0.00907975    0.002753426      0x02             0x00                                                                                  0                        0                      0                                                               0x0068    0x5000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                                                                                                                                          "application/octet-stream"                                                                                                                                       "_softw_90_update_u7iavi2512u2511z7.bin_8_B_1_0"                                                                                                                                                                                                                                                                                                                                                                                                                                                     "Apache"
A     10       0x0400000000004000  1258594163.408285  1258594191.015208  27.606923  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105   07       "Private network"              49330    143.166.11.10   us       "Dell"                         64334    6        1514        3101         0            4268858       0         0         0           0           0       5.58724   0.01823444   0.1478493    54.84132   0         -0.3438787   -1          0x49b1    1           223         128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3392384642  1511        0               0               1511        4255056                437             0           8192          45402        0            64860        253             194            254                0.001981506   0x0416    0x0008      511           1536       0x0000003e  1460    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.078742       1e-06             0.67088           0.03865783        0.04056465           0.07920299    0                0x42             0x02     9                                                                            0                        0                      1         "/video/R79733.EXE"                                   0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0
B     10       0x0400000000004001  1258594163.487027  1258594185.427506  21.940479  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10   us       "Dell"                         64334    192.168.1.105   07       "Private network"              49330    6        3101        1514         4268858      0             0         1380      1376.607    60.23097    0       0.67109   0.007075286  0.02745561   141.3369   194565.4  0.3438787    1           0x0c51    1           6365        111       111       0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3177226489  2866        5233476         27              3099        0                      0               28980       8192          64860        8192         64860        0               1              1                  0             0x031b    0x02c4      1             4          0x00000016  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000461       2e-06             5.587702          0.004777641       0.1447472            0.04343547    0.1503238        0x02             0x02     9                                                                            0                        0                      1         "/video/R79733.EXE"                                   0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0
A     9        0x0400000000004000  1258594162.928342  1258594185.618346  22.690004  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105   07       "Private network"              49329    143.166.11.10   us       "Dell"                         21       6        11          10           92           1231          0         24        8.363636    8.41835     0       21.78007  2.062728     5.945361     0.484795   4.054649  0.04761905   -0.8609222  0x0811    1           2328        128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  2427598871  10          92              0               10          1231                   0               24          8192          62176.56     8192         64860        8               1              9                  0             0x001a    0x0000      1             4          0x00000016  1460    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.080252       0.077494          21.58924          2.22962           6.122048             0.08095       0                0x02             0x09     10           USER;PASS;TYPE;PASV;SIZE;RETR                                   1           "anonymous"  1           "IEUser@"  2         "I";"/video/R79733.EXE"                               0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0
B     9        0x0400000000004001  1258594163.008594  1258594185.427515  22.418921  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10   us       "Dell"                         21       192.168.1.105   07       "Private network"              49329    6        10          11           1231         92            0         950       123.1       241.7142    0       21.78333  2.241892     6.179659     0.4460518  54.90898  -0.04761905  0.8609222   0x0811    2732        26560       239       239       0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  365320932   10          1207            0               10          92                     0               971         4140          4207.147     4140         4232         0               6              1                  0             0x021a    0x0000      1             2          0x00000014  1380    0      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000698       0.00045           0.194089          0.04303964        0.07018124           2.27266       6.122451         0x02             0x09     10                                          220;331;230;200;227;213;125;226  0                        0                      1         "125 Data connection already open; Transfer startin"  0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0

As we ran t2 with the -s option, a packet file was also created.

tcol ~/results/nudel_packets.txt

%pktNo  flowInd  flowStat            time               pktIAT     pktTrip    flowDuration  numHdrs  hdrDesc       ethVlanID  srcMac             dstMac             ethType  srcIP           srcIPCC  srcIPOrg                     srcPort  dstIP           dstIPCC  dstIPOrg                     dstPort  l4Proto  pktLen  l7Len  ipToS  ipID   ipIDDiff  ipFrag  ipTTL  ipHdrChkSum  ipCalChkSum  l4HdrChkSum  l4CalChkSum  ipFlags  ip6HHOptLen  ip6HHOpts  ip6DOptLen  ip6DOpts  ipOptLen  ipOpts  seq         ack         seqMax      seqDiff  ackDiff  seqLen  ackLen  seqFlowLen  ackFlowLen  tcpMLen  tcpBFlgt  tcpFStat  tcpFlags  tcpAnomaly  tcpWin  tcpWS  tcpMSS  tcpTmS  tcpTmER  tcpOptLen  tcpOpts                                                                                                                                      tcpStatesAFlags  ftpStat  l7Content
1       1        0x0400000000004000  1258544216.908284  0.000000   0.000000   0.000000      3        eth:ipv4:tcp             00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800   192.168.1.104   07       Private network              1260     198.189.255.75  us       California State University  80       6        66      0      0x00   1571   0         0x4000  128    0x6c8b       0x6c8b       0x44f1       0x44f1       0x0040   0                       0                     0                 1942699277  0           1942699277  0        0        0       0       0           0           0        0          0x0010    0x0002    0x0000      65535   0      1460    0       0        8          0x02;0x04;0x05;0xb4;0x01;0x01;0x04;0x02                                                                                                      0x00             0x00
2       1        0x0400000000004001  1258544216.915576  0.000000   0.007292   0.000000      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800   198.189.255.75  us       California State University  80       192.168.1.104   07       Private network              1260     6        62      0      0x00   0      0         0x4000  57     0xb9ae       0xb9ae       0x1902       0x1902       0x0040   0                       0                     0                 4272559795  1942699278  4272559795  0        0        0       0       0           0           0        0          0x0010    0x0212    0x0000      5840    0      1380    0       0        8          0x02;0x04;0x05;0x64;0x01;0x01;0x04;0x02                                                                                                      0x00             0x00
3       1        0x0400000000004000  1258544216.916026  0.007742   0.000450   0.007742      3        eth:ipv4:tcp             00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800   192.168.1.104   07       Private network              1260     198.189.255.75  us       California State University  80       6        64      0      0x00   1572   1         0x4000  128    0x6c92       0x6c92       0x5c46       0x5c46       0x0040   0                       0                     0                 1942699278  4272559796  1942699278  1        0        0       0       0           0           0        0          0x0011    0x0010    0x0000      65535   0      1460    0       0        0                                                                                                                                                       0x00             0x00
4       1        0x0400000000004000  1258544216.929764  0.013738   0.014188   0.021480      3        eth:ipv4:tcp             00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800   192.168.1.104   07       Private network              1260     198.189.255.75  us       California State University  80       6        377     319    0x00   1573   1         0x4000  128    0x6b52       0x6b52       0x4f1a       0x4f1a       0x0040   0                       0                     0                 1942699278  4272559796  1942699278  0        0        0       0       0           0           319      319        0x0011    0x0018    0x0000      65535   0      1460    0       0        0                                                                                                                                                       0x00             0x00     GET /softw/90/update/u7avi1777u1705ff.bin HTTP/1.1\r\nUser-Agent: AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA=\r\nHost: aa.avg.com\r\nAccept: */*\r\nAccept-Encoding: identity,deflate,gzip\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nx-avg-id:78-175947826\r\n\r\n
5       1        0x0400000000004001  1258544216.936827  0.021251   0.007063   0.021251      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800   198.189.255.75  us       California State University  80       192.168.1.104   07       Private network              1260     6        54      0      0x00   42177  42177     0x4000  57     0x14f5       0x14f5       0x41e7       0x41e7       0x0040   0                       0                     0                 4272559796  1942699597  4272559796  1        319      0       319     0           319         0        0          0x0811    0x0010    0x0000      6432    0      1380    0       0        0                                                                                                                                                       0x00             0x00
6       1        0x0400000000004001  1258544216.937559  0.000732   0.007795   0.021983      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800   198.189.255.75  us       California State University  80       192.168.1.104   07       Private network              1260     6        1434    1380   0x00   42178  1         0x4000  57     0x0f90       0x0f90       0x266d       0x266d       0x0040   0                       0                     0                 4272559796  1942699597  4272559796  0        0        0       0       0           319         1380     1380       0x0811    0x0010    0x0000      6432    0      1380    0       0        0                                                                                                                                                       0x00             0x00     HTTP/1.1 200 OK\r\nServer: Apache\r\nETag: "0210a9516dd34abc481683f877bd8680:1258533754"\r\nLast-Modified: Wed, 18 Nov 2009 07:55:25 GMT\r\nAccept-Ranges: bytes\r\nContent-Length: 95323\r\nContent-Type: application/octet-stream\r\nDate: Wed, 18 Nov 2009 11:39:50 GMT\r\nConnection: keep-alive\r\n\r\nMZ AVG7 UpdateBin grp(avi:1777)dep(avi:1705)tm(0911180855)pri(2)..7TW;........"....m.b...YbU..&..6.P.B.....jx.\n.n..%....g...8......c....X.c.sO..M............Y.7|......q........w/mb.D#...:.`.H|..(.:...wjA/...u....C{.]..7.y..8..v....n.5..L.k..U>&te...-.....a...`..n. h.....0.......9Ig.s..7^.)..,........ .R..+...f ...xg..xq....;1...F.|....)..*..~.%.I.o.*......)...P...w.V.q....41....h...w%o..,Ha;.~}..#!.p....{..w.=A.0...8..IB.;.*...]..w.@..%F[L9(.. ..`..Iq...'......4.&..........Gz0S}`...s.....s...6\).4(..x.J..[do...w./..m..[.X.D...z\.. ..F...\nA[....O_...."..te..|b..".......\t..e-..i.q....<&h....SKz.gR.+.<1....n........|...-...B..?..".../.g.I@..m[s....iu3$.t\tL...`...D$..eff..7(.L.\V_..HR!.X.........A#....=...K.[.>..CO.2J...R...k.k.p..ME...\}.v..l_.D\t...D...;c......0~3:A......i..7X&..].@.......k?..Qn........,c.`..K.\t.B.M........~\.....>..|._. ...W.YP.....N...u.....s@:..Z.z..n.."B..Q.M.9..D[.c.z.l...z.G....l..6.yPJ.8.........Q.eE.....oPK.'.s. ..(....+..3........."q...d.....v....@......q..+. _YK.`.Zn.c..a..E.q...cI......c....\r0..\n.... ]p..Z=.{./Iz..'..<.d...9...]:...P.}v<...9.h...T9cf../<..U.L
...

Have a look into it and see what you can find. Have fun!

Another method is the alarm support mode and the findexer which we will discuss in the following chapters.

Extract pcaps using the FL_ALARM bit

If t2 -e option is not present, pcapd extracts all packets if a plugin sets an internal flag in flow status using the macro:

T2_SET_STATUS(flowP, FL_ALARM)

We will have an example in the Alarm controlled packet extraction chapter of the Alarm mode tutorial and one in the regex based pcap extraction tutorial.

In general, findexer is the fastest and most efficient way to extract flows to pcaps, as you only need to run it once, and then you can change your extraction rule as often as you want. pcapd always requires a rerun of t2.

The superior way: findexer plugin

If you do not want to build an alarming tool for anomalies which extracts packets from on plugin request or if you have a stack of questions to a large pcap, and you do not want to rerun T2 all over again several times, the findexer plugin is the tool of choice.

So compile findexer in your plugin repository and rerun T2 on the original pcap.

t2build findexer

t2 -r ~/data/faf-exercise.pcap -w ~/results/

================================================================================
Tranalyzer 0.9.0(Anteater), Tarantula. PID: 7098
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: protoStats, 0.9.0
    02: basicFlow, 0.9.0
    03: basicStats, 0.9.0
    04: tcpFlags, 0.9.0
    05: tcpStates, 0.9.0
    06: ftpDecode, 0.9.0
    07: httpSniffer, 0.9.0
    08: txtSink, 0.9.0
    09: findexer, 0.9.0
[INF] IPv4 Ver: 5, Rev: 09082023, Range Mode: 0, subnet ranges loaded: 481438 (481.44 K)
[INF] IPv6 Ver: 5, Rev: 09082023, Range Mode: 0, subnet ranges loaded: 41486 (41.49 K)
Processing file: /home/wurst/data/faf-exercise.pcap
Link layer type: Ethernet [EN10MB/1]
...

findexer produces a binary ~/results/faf-exercise_flows.xer file, which contains all references between flow indexes and packets. Move to your results bash window or move to the results directory and do a ls then you spot it.

ls ~/results

faf-exercise_flows.txt  faf-exercise_flows.xer  faf-exercise_headers.txt  faf-exercise_icmpStats.txt  faf-exercise_protocols.txt  nudel  nudel_pcapd.pcap  nudel_flows.txt
nudel_headers.txt   nudel_icmpStats.txt   nudel_protocols.txt

Now invoke the tawk command which selects the flows from the pcapd section and add the -x option to produce a pcap.

tawk -H -x knoedel.pcap '{ if (bitsanyset($ftpStat, 0x0f) || bitsanyset($httpAFlags, 0x5000)) { print $flowInd } else { split($httpImg_Vid_Aud_Msg_Txt_App_Unk, A, "_"); if (A[1] > 0) print $flowInd } }' ~/results/faf-exercise_flows.txt

[INFO] Processing findexer : /home/wurst/results/faf-exercise_flows.xer
[INFO] Processing pcap 1/1 : /home/wurst/data/faf-exercise.pcap
[INFO] Extracted 4959 packets.

And ls will show you the knoedel.pcap file.

ls ~/results

faf-exercise_flows.txt  faf-exercise_flows.xer  faf-exercise_headers.txt  faf-exercise_icmpStats.txt  faf-exercise_protocols.txt  knoedel.pcap  nudel  nudel_pcapd.pcap   nudel_flows.txt
nudel_headers.txt   nudel_icmpStats.txt   nudel_protocols.txt

You may now run T2 on knoedel.pcap and you will see only the selected flows as in the pcapd section. The main difference is that you can run another tawk command with a different question and you will get right away a new pcap without re-invoking T2. E.g. a troubleshooting question: Select all flows where the window size was reported to 0, so the receiver could not process more packets anymore. Use the TCP Window size threshold ratio variable: tcpWinSzThRt from tcpFlags (threshold is 1 by default). I leave this exercise to the reader.

findexer and Wireshark export

There are people who like to work with Wireshark or tshark instead of t2 packet files. Fair enough, a bit more complicated if you have a lot of data.

So tawk comes to the rescue. Simply invoke tawk with the same question as above, but this time, add the -k option:

tawk -H -k '{ if (bitsanyset($ftpStat, 0x0f) || bitsanyset($httpAFlags, 0x5000)) { print $flowInd } else { split($httpImg_Vid_Aud_Msg_Txt_App_Unk, A, "_"); if (A[1] > 0) print $flowInd } }' ~/results/faf-exercise_flows.txt

[INFO] Processing findexer : /home/wurst/results/faf-exercise_flows.xer
[INFO] Processing pcap 1/1 : /home/wurst/data/faf-exercise.pcap
 ** (wireshark:9617) 14:26:09.519872 [Capture MESSAGE] -- Capture Start ...
[INFO] Extracted 4959 packets.
 ** (wireshark:9617) 14:26:09.770420 [Capture MESSAGE] -- Capture started
 ** (wireshark:9617) 14:26:09.770512 [Capture MESSAGE] -- File: "/tmp/wireshark_-B89ZG1.pcapng"
 ** (wireshark:9617) 14:26:09.917255 [Capture MESSAGE] -- Capture stopped.

If Wireshark complains that you do not have rights as a user then use the following command

sudo gpasswd -a YOUR_USERNAME wireshark

And log off and on again. Then try the tawk above again and you will have Wireshark open with knoedel.pcap.

tawk pcap export to Wireshark

How cool is that?

Management of humongous flow files: tawk ffsplit

Very large pcaps produce large flow files, if you do not aggregate enough or if you have too many plugins. Right! There is a way to split these files. One way is to use the -W option in T2 which produces many smaller flow files instead of one large file.

If you are interested in segregating several protocols, that is possible with tawk and the ffsplit function:

tawk -d ffsplit

ffsplit(s, k, h):

  Split the input file into smaller more manageable files.

  The files to create can be specified as argument to the function (one comma
  separated string) (See `s`).

  If no argument is specified, creates one file per column whose name ends
  with Stat, e.g., dnsStat, and one for pwxType (pw) and covertChannels (cc)

  `s` Valid arguments for 's' are:
    - arp        - bgp        - dhcp        - dns        - ftp        - http
    - icmp       - igmp       - irc         - modbus     - nat        - ntp
    - ospf       - pop        - radius      - sctp       - smb        - smtp
    - ss         - stp        - syslog      - tftp       - voip       - vrrp

    - pw (cleartext passwords)
    - cc (covert channels)

  `h` Valid arguments for 'h' are:
    - if h is empty or omitted: keep the default columns
        default = "dir,flowInd,timeFirst,timeLast,duration,ethVlanID,
                srcIP,srcIP4,srcIP6,dstIP,dstIP4,dstIP6,srcPort,dstPort,l4Proto"
    - if h is a non-existent column, e.g., 0: keep only relevant fields
    - if h starts with '+', e.g., "+flowStat,hdrDesc", keep default fields
      and the ones specified
    - else keep fields specified in h, e.g., "srcIP,dstIP"

  Parameters:
    - s: a comma separated list of files to create (see `s`).
    - k: 0: keep all columns, 1: keep only relevant columns
    - h: comma separated list of fields to keep in each file (see `h`)

  Dependencies:
    - None

  Examples:
    - tawk 'ffsplit()' file.txt
    - tawk 'ffsplit("dns")' file.txt
    - tawk 'ffsplit("cc,dns,http")' file.txt
    - tawk 'ffsplit("", 1)' file.txt
    - tawk 'ffsplit("dns,http", 1, "+flowStat,hdrDesc")' file.txt
    - tawk 'ffsplit("", 1, "srcIP,dstIP")' file.txt

Let’s try the simplest way first, extract all protocols in separate flow files:

tawk 'ffsplit()' ~/results/faf-exercise_flows.txt

ls ~/results

faf-exercise_flows_ftp.txt   faf-exercise_flows_icmp.txt  faf-exercise_flows.xer    faf-exercise_icmpStats.txt  knoedel.pcap  nudel_pcapd.pcap
faf-exercise_flows_http.txt  faf-exercise_flows.txt       faf-exercise_headers.txt  faf-exercise_protocols.txt  nudel

So you see that now for each existing protocol a different flow file is produced. Where is the SMTP? Right, we didn’t load the smtpDecode plugin. Was your home work, remember? Let’s look into FTP:

tcol ~/results/faf-exercise_flows_ftp.txt

%dir  flowInd  flowStat            timeFirst          timeLast           duration    numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP          srcIPCC  srcIPOrg           srcPort  dstIP          dstIPCC  dstIPOrg           dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT       stdIAT      pktps       bytps     pktAsm      bytAsm      tcpFStat  ipMindIPID  ipMaxdIPID  ipMinTTL  ipMaxTTL  ipTTLChg  ipToS  ipFlags  ipOptCnt  ipOptCpCl_Num    ip6OptCntHH_D  ip6OptHH_D             tcpISeqN    tcpPSeqCnt  tcpSeqSntBytes  tcpSeqFaultCnt  tcpPAckCnt  tcpFlwLssAckRcvdBytes  tcpAckFaultCnt  tcpBFlgtMx  tcpInitWinSz  tcpAveWinSz  tcpMinWinSz  tcpMaxWinSz  tcpWinSzDwnCnt  tcpWinSzUpCnt  tcpWinSzChgDirCnt  tcpWinSzThRt  tcpFlags  tcpAnomaly  tcpOptPktCnt  tcpOptCnt  tcpOptions  tcpMSS  tcpWS  tcpMPTBF  tcpMPF  tcpMPAID  tcpMPdssF  tcpTmS  tcpTmER  tcpEcI  tcpUtm    tcpBtm    tcpSSASAATrip  tcpRTTAckTripMin  tcpRTTAckTripMax  tcpRTTAckTripAve  tcpRTTAckTripJitAve  tcpRTTSseqAA  tcpRTTAckJitAve  tcpStatesAFlags  ftpStat  ftpCDFindex  ftpCC                          ftpRC                            ftpNumUser  ftpUser      ftpNumPass  ftpPass    ftpNumCP  ftpCP                                                 smtpStat  smtpCC  smtpRC  smtpUsr  smtpPW  smtpSANum  smtpESANum  smtpERANum  smtpSA  smtpESA  smtpERA  httpStat  httpAFlags  httpMethods  httpHeadMimes  httpCFlags  httpGet_Post  httpRSCnt  httpRSCode  httpURL_Via_Loc_Srv_Pwr_UAg_XFr_Ref_Cky_Mim  httpImg_Vid_Aud_Msg_Txt_App_Unk  httpHosts  httpURL  httpMimes  httpCookies  httpImages  httpVideos  httpAudios  httpMsgs  httpAppl  httpText  httpPunk  httpBdyURL  httpUsrAg  httpXFor  httpRefrr  httpVia  httpLoc  httpServ  httpPwr
A     36       0x0400000000004000  1258594163.408285  1258594191.015208  27.606923   1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        1514        3101         0            4268858       0         0         0           0           0       5.58724   0.01823444   0.1478493   54.84132    0         -0.3438787  -1          0x49b1    1           223         128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3392384642  1511        0               0               1511        4255056                437             0           8192          45402        0            64860        253             194            254                0.001981506   0x0416    0x0008      511           1536       0x0000003e  1460    4      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.078742       1e-06             0.67088           0.03865783        0.04056465           0.07920299    0                0x42             0x02     35                                                                           0                        0                      1         "/video/R79733.EXE"                                   0x00                                       0          0           0                                     0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0
B     36       0x0400000000004001  1258594163.487027  1258594185.427506  21.940479   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        3101        1514         4268858      0             0         1380      1376.607    60.23097    0       0.67109   0.007075286  0.02745561  141.3369    194565.4  0.3438787   1           0x0c51    1           6365        111       111       0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3177226489  2866        5233476         27              3099        0                      0               28980       8192          64860        8192         64860        0               1              1                  0             0x031b    0x02c4      1             4          0x00000016  1380    1      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000461       2e-06             5.587702          0.004777641       0.1447472            0.04343547    0.1503238        0x02             0x02     35                                                                           0                        0                      1         "/video/R79733.EXE"                                   0x00                                       0          0           0                                     0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0
A     35       0x0400000000004000  1258594162.928342  1258594185.618346  22.690004   1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49329    143.166.11.10  us       "Dell"             21       6        11          11           92           1231          0         24        8.363636    8.41835     0       21.78007  2.062728     5.945361    0.484795    4.054649  0           -0.8609222  0x0811    1           2328        128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  2427598871  10          92              0               10          1231                   0               24          8192          62176.56     8192         64860        8               1              9                  0             0x001a    0x0000      1             4          0x00000016  1460    1      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.080252       0.077494          306.0649          29.85102          83.48595             0.08095       0                0x02             0x09     36           USER;PASS;TYPE;PASV;SIZE;RETR                                   1           "anonymous"  1           "IEUser@"  2         "I";"/video/R79733.EXE"                               0x00                                       0          0           0                                     0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0
B     35       0x0400000000004001  1258594163.008594  1258594491.683288  328.674694  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10  us       "Dell"             21       192.168.1.105  07       "Private network"  49329    6        11          11           1231         92            0         950       111.9091    232.9224    0       306.2558  29.87952     83.53862    0.03346774  3.745345  0           0.8609222   0x0811    2732        26560       239       239       0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  365320932   11          1231            0               11          92                     0               971         4140          4214.603     4140         4232         0               6              1                  0             0x061e    0x0000      1             2          0x00000014  1380    1      0x0000    0x00    0         0x00       0       0        0       0.000000  0.000000  0.000698       0.00045           0.194089          0.04303964        0.07018124           29.89405      83.48597         0x42             0x09     36                                          220;331;230;200;227;213;125;226  0                        0                      1         "125 Data connection already open; Transfer startin"  0x00                                       0          0           0                                     0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0

Yep all FTP. Does that help you? Give us feedback please.

If somebody is interested in why it is so easy to produce all kinds of functionality with tawk, the answer is in the fextractor which does all the work of extracting all packets belonging to a given flow, using the .xer file.

Working on findexed pcaps: fextractor

Imagine you have only a distinct selection of pcaps to analyze, and they are big, so that you do not want to run Tranalyzer every time you change your question, or you select certain flows. And imagine you want to script everything yourself without using tawk -k because you want to do your own thing. Fair enough, then fextractor is your tool of choice. Note that tawk -k uses fextractor as well.

Let’s have a look at the help first

fextractor -h

Usage: fextractor -r INPUT[:start][,end] (-w OUTPUT | -n) [OPTIONS]... \
            [[DIR@]FLOWINDEX[:start][,end]]...

Extract the flows FLOWINDEX using the _flows.xer INTPUT generated by Tranalyzer2 findexer plugin.
Alternatively use a list of findexer files generated by Tranalyzer2 -W option from index start
to end. The extracted flows are written to the OUTPUT pcap.

An optional packet range can be provided on each command line FLOWINDEX to only extract packets
in the range [start, end] of this flow. If start or end are omitted, they are replaced by,
respectively, the first and the last available packets in the flow. The FLOWINDEX can also
optionally be prefixed with a direction A or B, by default both directions are extracted.

OPTIONS:
  -r INPUT[:start][,end]
            either read packet indexes from a single _flows.xer file named INPUT
            or read packet indexes from multiple _flows.xer files prefixed by INPUT
            and with suffix in range [start, end]. If start or end are omitted,
            they are replaced by, respectively, first and last available XER files.
  -w OUTPUT write packets to pcap file OUTPUT
            OUTPUT "-" means that the PCAP is written to stdout.
  -f        overwrite OUTPUT if it already exists
  -n        print oldest PCAP still available, its first packet timestamp and exit
  -h        print this help message
  -i FILE   read flow indexes from FILE. FILE can either be in _flows.txt format
            (flow index in 2nd tab-separated column), or have one flow index per line.
            FILE "-" means that flows are read from stdin.
  -b        by default when FILE is in _flows.txt format, only directions present in
            it are extracted, this option force both directions to be extracted even if
            only the A or B direction is present in the flow file.
  -s N      skip the first N PCAPs
  -p DIR    search pcaps in DIR
            should only be set if pcaps were moved since Tranalyzer2 was run

The -r option allows you to process any pcap which is indexed by findexer. If you have several files which are indexed, e.g. nudel1, nudel2, … then use the colon notation. The -w option denotes the location and name where the extracted pcap should be written. The -i option denotes the flow index file. From our earlier experiment with pcapd above we still have our flow index file: nudel. So move to your results folder and invoke fextractor using the already generated findexer faf-exercise_flows.xer file. The -b option enables to extract also the opposite flow, which might not match the extraction condition. Nevertheless, this is very useful for forensics purposes to have the full traffic being involved in one pcap.

fextractor -i ~/results/nudel -r ~/results/faf-exercise_flows.xer -w ~/results/nudelxer.pcap

ls ~/results

faf-exercise_flows.txt  faf-exercise_flows.xer  faf-exercise_headers.txt  faf-exercise_icmpStats.txt  faf-exercise_protocols.txt
nudel  nudel.pcap  nudelxer.pcap

Now you may change the ~/results/nudel file and re-invoke the same fextractor command without running T2 ever again, if you are only interested in the same pcap.

fextractor as flow based pcap slicer

If your pcap is really large, and you want to split it into smaller chunks we advice to use the -c option of tcpdump. But if you want to sort the packets according to their flow affiliation into separate packet files the fextractor is your friend. Below is a script which creates small pcaps, where each pcap contains one flow.

#!/usr/bin/env bash

if [ -z "$1" ]; then
   echo "Usage: ./t2pef numFlows"
   exit 1
fi

shopt -s expand_aliases
source "$HOME/tranalyzer2/scripts/t2_aliases"

for ((i = 1; i <= $1; i++)) do
   echo $i > ~/results/nudel
   fextractor -i ~/results/nudel -r ~/results/faf-exercise_flows.xer -w ~/results/nudelxer$i.pcap
done

Note that a substantial number of pcap files will be created which might disturb your OS. The pcapd plugin might be faster here.

pcap generation and upscaling

So you have a pcap at low speed and want to convert it to high speed with the same statistic properties. pcapd will come to the rescue.

...
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */

#define PD_MODE_PKT  0  // 0: all packets, 1: activate packet range selection
#define PD_STRTPKT   1  // Packet at which processing starts (PD_MODE_PKT=1)
#define PD_ENDPKT    10 // 0: End of flow, > 0: Packet at which processing ends (PD_MODE_PKT=1)

#define PD_MODE_IN   0 // 0: extract flows listed in input file (if -e option was used),
                       //    or extract flows if alarm bit is set (if -e option was not used)
                       // 1: dump all packets
#define PD_LBSRCH    1 // search 1: binary, 0: linear
#define PD_EQ        1 // whether to dump matching (1) or non-matching (0) flows
#define PD_OPP       0 // if PD_MODE_IN extract also the opposite flow, 0: don't

#define PD_MODE_OUT  0 // 0: one pcap,
                       // 1: one pcap per flow

#define PD_SPLIT     1 // whether or not to split output file (-W option)

#define PD_TSHFT     0 // Timeshift
#define PD_TTSFTS    0 // Add value to secs
#define PD_TTSFTNMS  1 // Add value to usec/nsec (depends on TSTAMP_PREC in tranalyzer.h)

#define PD_MACSHFT   0 // MAC shift last byte
#define PD_MACSSHFT  1 // Add value to src MAC
#define PD_MACDSHFT  1 // Add value to dst MAC

#define PD_VLNSHFT   0 // VLAN ID shift
#define PD_VLNISHFT  1 // Add value to inner VLAN ID

#define PD_IPSHFT    0                  // Add to src/dst IP
#define PD_IP4SHFT   0x00000001         // Add value to IPv4 32 bit
#define PD_IP6SHFT   0x0000000000000001 // Add value to IPv6 shift last 64 bit

#define PD_TTLSHFT   0    // 0: no TTL change, 1: TTL shift, 2: random shift
#define PD_TTL       8    // sub value from TTL
#define PD_TTLMOD    128  // TTL modulo

#define PD_CHKSUML3  0    // Correct checksum in IPv4 header

...
/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...

Conclusion

That’s it! Now reset the configuration of core and all plugins for later tutorials:

t2conf -a --reset

Have fun experimenting!