Plugin pcap extraction
Contents
Introduction
As detailed in the pcap extraction tutorial,
T2 is capable to extract flows into pcaps using the
pcapd plugin.
The regex_pcre is an example for this operation
regex_pcre flow extraction.
In order to implement this feature in your own plugin, specialized macros are supplied, which will be detailed in this tutorial.
You do not need to enable the ALARM_MODE it also works in the normal flow mode.
Getting started
Create folders for your data and results
If you have not created a separate data and results directory yet, please do it now. This will greatly facilitate your workflow:
mkdir ~/data ~/results
Reset tranalyzer2 and the plugins configuration
If you have followed the other tutorials, you may have modified some of the core and plugins configuration. To ensure your results match those in this tutorial, make sure to reset everything:
t2conf -a --reset
You can also clean all build files:
t2build -a -c
Empty the plugin folder
To ensure we are not left with some unneeded plugins or plugins which were built using different core configuration, it is safer to empty the plugins folder:
t2build -e -y
Are you sure you want to empty the plugin folder '/home/user/.tranalyzer/plugins' (y/N)? yes
Plugin folder emptied
Download the PCAP file
The PCAP file used in this tutorial can be downloaded here:
Please save it in your ~/data folder:
wget --no-check-certificate -P ~/data https://tranalyzer.com/download/data/faf-exercise.pcap
Build tranalyzer2 and the required plugins
For this tutorial, we will need to build the core (tranalyzer2) and the following plugins:
As you may have modified some of the automatically generated files, it is safer to use the -r and -f options.
...
BUILDING SUCCESSFUL
Source code
In this tutorial, we will extend tcpWin12.tar.gz, the final version of the previous tutorial (Plugin force mode).
If you are impatient, you can download the final version of the tcpWin plugin we will develop in this tutorial.
To use one of those plugins, just unpack it in the plugins folder of your T2 installation.
tranpl
tar -xf ~/Downloads/tcpWin13.tar.gz
And let t2_aliases know about it:
source "$T2HOME/scripts/t2_aliases"
Implementing the FL_ALARM capability
Open tcpWin.c in an editor and move to the t2OnLayer4() callback.
The T2_SET_STATUS() line was already added in the Plugin Alarm Mode tutorial, so we do not need to add it again. Now, we want the PCAP extraction to work
independently of the ALARM_MODE. So we remove the #if ALARM_MODE == 1 and #endif // ALARM_MODE == 1 pragmas before and after T2_SET_STATUS() (line marked by // <--).
tcpWin
vi src/tcpWin.c
...
void t2OnLayer4(packet_t *packet, unsigned long flowIndex) {
flow_t * const flowP = &flows[flowIndex];
if (flowP->l4Proto != L3_TCP) { // process only TCP
TCPWIN_SPKTMD_PRI_NONE(); // Packet mode
return; // go back to core
}
// only 1. frag packet will be processed
if (!t2_is_first_fragment(packet)) {
TCPWIN_SPKTMD_PRI_NONE(); // Packet mode
return; // go back to core
}
tcpWinFlow_t * const tcpWinFlowP = &tcpWinFlows[flowIndex];
const tcpHeader_t * const tcpHeader = TCP_HEADER(packet);
const uint32_t tcpWin = ntohs(tcpHeader->window);
if (tcpWin < TCPWIN_THRES) { // is the window size below the threshold?
tcpWinFlowP->winThCnt++; // count the packet / flow
winThCntG++; // increment global packet counter
tcpWinFlowP->stat |= TCPWIN_STAT_THU; // set the status bit
tcpWinStat |= tcpWinFlowP->stat; // Aggregate all packet flags
T2_SET_STATUS(flowP, FL_ALARM); // <-- Set the alarm bit in flow and global
#if FORCE_MODE == 1
if (tcpWinFlowP->winThCnt > 1) T2_RM_FLOW(flowP);
#endif // FORCE_MODE == 1
}
// Packet mode
if (sPktFile) {
fprintf(sPktFile,
"%" PRIu32 /* tcpWinSize */ SEP_CHR
"%" PRIu32 /* tcpWinThPktCnt */ SEP_CHR
, tcpWin, tcpWinFlowP->winThCnt);
}
}
...Now pcapd will extract all packets of a flow direction where the FL_ALARM is set.
If you want to have the global FL_ALARM status including flow and alarm count shown
as a warning in the end report, then move the T2_REPORT_ALARMS() line in the t2OnFlowTerminate() callback outside the #if ALARM_MODE == 1 and #endif // ALARM_MODE == 1 pragmas.
vi src/tcpWin.c
...
void t2OnFlowTerminate(unsigned long flowIndex, outputBuffer_t *buf) {
const tcpWinFlow_t * const tcpWinFlowP = &tcpWinFlows[flowIndex];
bSFlow_t * const bSFlowP = &bSFlow[flowIndex];
float f = 0.0;
if (bSFlowP->numTPkts) {
f = (float)tcpWinFlowP->winThCnt/(float)bSFlowP->numTPkts;
}
OUTBUF_APPEND_U8(buf, tcpWinFlowP->stat);
OUTBUF_APPEND_U8(buf, tcpWinFlowP->ttl);
OUTBUF_APPEND_U32(buf, tcpWinFlowP->tcpWinInit);
OUTBUF_APPEND_U32(buf, tcpWinFlowP->winThCnt);
OUTBUF_APPEND_FLT(buf, f);
flow_t * const flowP = &flows[flowIndex];
if (!FLOW_IS_IPV4(flowP) || !PROTO_IS_TCP(flowP)) return; // IPv4 and TCP only
if (tcpWinFlowP->winThCnt == 0 || bSFlowP->numTPkts < TCPWIN_MINPKTS) return;
#if ALARM_MODE == 1
winAlarms += tcpWinFlowP->winThCnt; // global alarm count
winThFlows++; // count number of alarms flows
#endif // ALARM_MODE == 1
T2_REPORT_ALARMS(tcpWinFlowP->winThCnt); // <-- report alarm to the core
const int wzi = gwz.wzi; // store element count in const local variable, makes the compiler happy
if (wzi >= TCPWIN_MAXWSCNT) return; // If array full, stop saving
int i;
for (i = 0; i < wzi; i++) {
if (gwz.wzip[i].IPv4.s_addr == flowP->srcIP.IPv4.s_addr) break; // does IP exist?
}
if (f > gwz.wzCnt[i]) { // only update if count is greater than the previous one
gwz.tcpCnt[i] = bSFlowP->numTPkts; // update packet count
gwz.wzCnt[i] = f; // update relative count
if (i == wzi) { // new one?
gwz.wzip[i] = flowP->srcIP; // save new IP
#if SUBNET_ON != 0 && (AGGREGATIONFLAG & SUBNET) == 0
gwz.sID[i] = flowP->subnetNrSrc; // save subnetID from core
#endif // SUBNET_ON != 0 && (AGGREGATIONFLAG & SUBNET) == 0
gwz.wzi++; // increment global window size counter
}
}
}
...After you edited the skeleton code you should compare your implementation with tcpWin13.tar.gz.
Recompile and run t2 on the pcap.
t2build tcpWin
t2 -r ~/data/faf-exercise.pcap -w ~/results/================================================================================ Tranalyzer 0.9.0 (Anteater), Cobra. PID: 91536, SID: 666 ================================================================================ [INF] Creating flows for L2, IPv4, IPv6 Active plugins: 01: basicFlow, 0.9.0 02: basicStats, 0.9.0 03: tcpStates, 0.9.0 04: tcpWin, 0.9.0 05: txtSink, 0.9.0 06: pcapd, 0.9.0 [INF] IPv4 Ver: 5, Rev: 09082023, Range Mode: 0, subnet ranges loaded: 481663 (481.66 K) [INF] IPv6 Ver: 5, Rev: 09082023, Range Mode: 0, subnet ranges loaded: 41533 (41.53 K) Processing file: /home/wurst/data/faf-exercise.pcap Link layer type: Ethernet [EN10MB/1] Snapshot length: 65535 Dump start: 1258544215.37210000 sec (Wed 18 Nov 2009 11:36:55 UTC) Dump stop : 1258594491.683288000 sec (Thu 19 Nov 2009 01:34:51 UTC) Total dump duration: 50276.646078000 sec (13h 57m 56s) Finished processing. Elapsed time: 0.009388000 sec Finished unloading flow memory. Time: 0.009429000 sec Percentage completed: 100.00% Number of processed packets: 5902 (5.90 K) Number of processed bytes: 4993414 (4.99 M) Number of raw bytes: 4993414 (4.99 M) Number of pcap bytes: 5087870 (5.09 M) Number of IPv4 packets: 5902 (5.90 K) [100.00%] Number of A packets: 1986 (1.99 K) [33.65%] Number of B packets: 3916 (3.92 K) [66.35%] Number of A bytes: 209315 (209.31 K) [4.19%] Number of B bytes: 4784099 (4.78 M) [95.81%] Average A packet load: 105.40 Average B packet load: 1221.68 (1.22 K) -------------------------------------------------------------------------------- basicStats: Biggest L3 flow talker: 143.166.11.10 (US): 3101 (3.10 K) [52.54%] packets basicStats: Biggest L3 flow talker: 143.166.11.10 (US): 4268858 (4.27 M) [85.49%] bytes tcpStates: Aggregated tcpStatesAFlags=0x4a tcpWin: Aggregated tcpWinStat=0x01 tcpWin: Number of TCP winsize packets below threshold 1: 4 [0.07%] tcpWin: IP: 192.168.1.105, country: 07, org: Private network pcapd: number of packets extracted: 1467 (1.47 K) [24.86%] -------------------------------------------------------------------------------- Headers count: min: 3, max: 3, average: 3.00 Number of TCP packets: 5902 (5.90 K) [100.00%] Number of TCP bytes: 4993414 (4.99 M) [100.00%] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Number of processed flows: 72 Number of processed IPv4 flows: 72 [100.00%] Number of processed A flows: 36 [50.00%] Number of processed B flows: 36 [50.00%] Number of request flows: 36 [50.00%] Number of reply flows: 36 [50.00%] Total A/B flow asymmetry: 0.00 Total req/rply flow asymmetry: 0.00 Number of processed packets/flows: 81.97 Number of processed A packets/flows: 55.17 Number of processed B packets/flows: 108.78 Number of processed total packets/s: 0.12 Number of processed A+B packets/s: 0.12 Number of processed A packets/s: 0.04 Number of processed B packets/s: 0.08 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Number of average processed flows/s: 0.00 Average full raw bandwidth: 795 b/s Average full bandwidth : 792 b/s Max number of flows in memory: 18 [0.01%] Memory usage: 0.01 GB [0.12%] Aggregated flowStat=0x0402000000004000 [WRN] 4 alarms in 1 flows [1.39%] [INF] IPv4 flows [INF] IPAlarm
The plugin report states that 1467 packets were extracted of one flow.
Note the [WRN] which reports the number of alarms.
0x0402000000004000 describes the global aggregate flow status which T2_SET_STATUS() also sets.
Try a tawk -V to decode the flow status:
The flowStat column with value 0x0402000000004000 is to be interpreted as follows: bit | flowStat | Description ============================================================================= 14 | 0x0000 0000 0000 4000 | IPv4 flow 49 | 0x0002 0000 0000 0000 | Alarm mode & pcapd dumps packets from this flow to new pcap if not -e option 58 | 0x0400 0000 0000 0000 | IPv4 packet
See?!
Now look into the results directory:
ls ~/results
faf-exercise_flows.txt faf-exercise_headers.txt faf-exercise_pcapd.pcap
Note the file faf-exercise_pcapd.pcap, which represents the extracted packets belonging to
the flow where the FL_ALARM bit is set. To verify, select these flows using tawk.
tawk 'bitsanyset($flowStat, 0x0002000000000000)' ~/results/faf-exercise_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPOrg srcPort dstIP dstIPCC dstIPOrg dstPort l4Proto tcpStates tcpWinStat tcpWinThCnt
A 36 0x0402000000004000 1258594163.408285 1258594191.015208 27.606923 1 3 eth:ipv4:tcp 00:08:74:38:01:b4 00:19:e3:e7:5d:23 0x0800 192.168.1.105 07 "Private network" 49330 143.166.11.10 us "Dell" 64334 6 0x42 0x01 4Only one flow? Why? Look at the config of pcapd:
t2conf -G PD_OPP pcapd
PD_OPP = 0So set it to 1 and see whether you get the other flow, as homework.
If you are uneasy working with pcapd refer to the PCAP extraction and upscaling tutorial.
If you are just interested in pcap extraction, then you do not need to load txtSink.
It is advisable to use the -w option, otherwise t2 stores the pcap in the data folder, and you start desperately looking for your extracted pcap.
It happened to people ;-)
Conclusion
Have fun extracting pcaps!
You can download the final version of the tcpWin plugin.
The next tutorial will teach you how to implement timeout handlers.
See also
- Plugin programming cheatsheet
- The basics: your first flow plugin
- Plugin end report
- Plugin monitoring
- Plugin packet mode
- Plugin summary files
- Plugin geo labeling
- Plugin dependencies
- Plugin alarm mode
- Plugin force mode
- Plugin flow timeout
- Plugin sink
- Developing Tranalyzer plugins in C++
- Developing Tranalyzer plugins in Rust