Analyses using the SiLK tool suite can become quite complex, with several intermediate products created while isolating the behavior of interest. One tool that can aid in managing these products isrwfileinfo, which displays a variety of characteristics for each file format produced by the SiLK tool suite. See Figure4.5
for parameter information. Some of these characteristics are shown in Example 4.5. rwfileinfo has a --fields parameter to allow analysts to specify the characteristics they want to see, as shown in Command 2 of Example4.5. For most analysts, the three most important characteristics are the record count, file size, and command-line information. For flow record files, the record count is the number of flow records in the file. For files with variable-length records (indicated by a record-length of one) the count-records field does not reflect the number of records; instead it is the uncompressed size (in bytes) of the data section. Notably, count-records does not reflect the number of addresses in an IP set file. The file-size field displays the size of the file on disk. Thecommand-lines field shows the commands used to generate the file. Flow record files produced byrwfilter maintain an historical record that can be used to trace how and where a file was created. This information can be extracted using the rwfileinfo command. Example4.5shows an example of the results from severalrwfileinfo commands. This field consists of a list of commands in historical order. This list includesrwfilter and a variety of other tools that add and preserve the command history. One tool that does not preserve this history (in case it needs to be cleared for any reason) isrwcat. Example4.6shows how thecommand-lines field expands with progressive commands and how rwcat does not preserve this history information.
Anannotations characteristic is supported by the --note-add parameter of many tools in SiLK, as shown in Command 6 of Example4.6. Annotations can be displayed using rwfileinfo.
Whilerwfileinfo is often associated with flow record files, it can also show information on sets, bags, and pmaps (see Section4.4, Section4.5, and Section4.7, respectively). Example4.7displays how these files are handled. In Commands 1 and 2, a set and a bag for the example are created. Command 3 shows a full rwfileinfo result for the set file. Commands 5 through 7 show just the specific information for the set, bag, and pmap file, respectively. The pmap file was generated outside of these commands with an internal name example-pmap. If a prefix map was created without a mapname, rwfileinfo will return an empty result for the prefix-map-specific field.
4.1. MANIPULATING FLOW RECORD FILES 85
Example 4.3: rwsplit for Coarse Parallel Execution
<1>$ rwfilter --type =inweb , outweb --start - date =2009/4/1 \
--end - date =2009/4/30 --bytes -per - packet =45 - --pass = stdout \ | rwsplit --flow - limit =10000 -- basename = part
<2>$ s_list =( skip ) # keep track of files generated <3>$ for f in part *; do
n=$( basename "$f ") t=${n %.*}
rm -f $t{-miss ,- threat ,- casual ,- other }. rw
rwfilter $f -- anyset = mission . set --pass =$t - miss .rw --fail = stdout \
| rwfilter stdin -- anyset = threat . set --pass =$t - threat .rw --fail = stdout \ | rwfilter stdin -- anyset = casual . set --pass =$t - casual .rw &
s_list =(${ s_list [*]} $t{-miss ,- threat ,- casual }. rw) done
<4>$ wait
<5>$ echo "Part -name , mission , threat , casual "; \ prev =" "; \ for f in ${ s_list [*]}; do if [ "$f" = skip ]; then continue fi cur =${f% -*} if [ " $prev " != " " ]; then
if [ " $cur " != " $prev " ]; then echo echo -n "$cur , " fi else echo -n "$cur , " fi prev = $cur
echo -n $( rwfileinfo -- fields =count - records $f | tail -n1 \ | sed -e "s/^ *count - records *//")" , "
done ; \ echo
Part -name , mission , threat , casual part .00000000 , 531 , 5205 , 4264 , part .00000001 , 299 , 5076 , 4625 , part .00000002 , 1004 , 3157 , 5839 , part .00000003 , 141 , 2146 , 7713 , part .00000004 , 60, 9436 , 504 , part .00000005 , 48, 8798 , 1154 , part .00000006 , 174 , 8041 , 1785 , part .00000007 , 122 , 8949 , 929 , part .00000008 , 258 , 4507 , 5235 , part .00000009 , 197 , 8677 , 1126 , part .00000010 , 248 , 4092 , 2087 ,
Example 4.4: rwsplit to Generate Statistics on Flow Record Files
<1>$ rwfilter --type =in , inweb --start - date =2009/4/1 \
--end - date =2009/4/30 -- protocol =0 -255 --pass = stdout \ | rwsplit --flow - limit =1000 --sample - ratio =100 \
-- basename = sample --max - outputs =100 <2>$ echo -n >udpsample . txt
<3>$ for f in sample *; do \
rwstats $f -- values = records -- fields = protocol -- count =30 --top \ | grep "17|" \
| cut -f3 "-d|" >> udpsample . txt \ done
<4>$ sort -nr udpsample . txt >tmp . txt <5>$ echo -n " Max UDP %: "; \
head -n 1 tmp . txt
Max UDP %: 6.400000
<6>$ echo -n " Min UDP %: " ; \ tail -n 1 tmp . txt
Min UDP %: 0.300000
<7>$ echo -n " Median UDP %: "; \ head -n 50 tmp . txt \
| tail -n 1
Median UDP %: 0.300000
Example 4.5: rwfileinfo for Display of Flow Record File Characteristics
<1>$ rwfileinfo medfile .rw medfile .rw:
format (id) FT_RWIPV6ROUTING (0 x0c )
version 16
byte - order littleEndian
compression (id) none (0)
header - length 352 record - length 88 record - version 1 silk - version 3.5.1 count - records 15 file - size 1672 command - lines
1 rwfilter -- sensor =S0 --type =in , inweb --start - date =2009/4/20 T12 --end - date =2009/4/20 T13 -- protocol =0- --note - add = example --pass = stdout
2 rwfilter -- duration =1200 - --pass = slowfile .rw --fail = stdout stdin
3 rwfilter -- duration =60 -1199 --pass = medfile .rw --fail = fastfile .rw stdin
annotations
1 example
<2>$ rwfileinfo medfile .rw -- fields =count - records medfile .rw:
4.1. MANIPULATING FLOW RECORD FILES 87
Figure 4.5: Summary of rwfileinfo
rwfileinfo
Description Displays summary information about one or more SiLK files
Call rwfileinfo allflows.rw --fields=count-records,command-lines Parameters --fields Selects which summary information to display via number or name
(by default, all the available fields). Possible values include
# Field Description
1 format Binary file format indicator 2 version Version of file header
3 byte-order Byte order of words written to disk 4 compression Type of space compression used 5 header-length Number of bytes in file header
6 record-length Number of bytes in fixed-length records 7 count-records Number of records in the file unless record-
length=1
8 file-size Total number of bytes in the file on disk 9 command-lines List of stored commands that generated this
file
10 record-version Version of records in file
11 silk-version Software version of SiLK tool that produced this file
12 packed-file-info Information from packing process 13 probe-name Probe info for files created by flowcap 14 annotations List of notes
15 prefix-map Prefix map name and header version 16 ipset IP-set format information
17 bag Bag key and count information --summary Prints a summary of total files, file sizes, and records
Example 4.6: rwfileinfo for Showing Command History
<1>$ rwfileinfo slowfile .rw --field = command - lines slowfile .rw:
command - lines
1 rwfilter -- sensor = SEN --type =in , inweb --start - date =2009/4/20 T12 --end - date =2009/4/20 T13 -- protocol =0- --note - add = example --pass = stdout
2 rwfilter -- duration =1200 - --pass = slowfile .rw --fail = stdout stdin
<2>$ rwfilter slowfile .rw -- protocol =6 --dport =22 --pass = newfile .rw <3>$ rwfileinfo newfile .rw --field = command - lines
newfile .rw: command - lines
1 rwfilter -- sensor = SEN --type =in , inweb --start - date =2009/4/20 T12 --end - date =2009/4/20 T13 -- protocol =0- --note - add = example --pass = stdout
2 rwfilter -- duration =1200 - --pass = slowfile .rw --fail = stdout stdin
3 rwfilter -- protocol =6 --dport =22 --pass = newfile .rw slowfile .rw
<4>$ rwsort newfile .rw -- fields = sTime --output - path = sorted .rw <5>$ rwfileinfo sorted .rw --field = command - lines
sorted .rw: command - lines
1 rwfilter -- sensor = SEN --type =in , inweb --start - date =2009/4/20 T12 --end - date =2009/4/20 T13 -- protocol =0- --note - add = example --pass = stdout
2 rwfilter -- duration =1200 - --pass = slowfile .rw --fail = stdout stdin
3 rwfilter -- protocol =6 --dport =22 --pass = newfile .rw slowfile .rw
4 rwsort -- fields = sTime --output - path = sorted .rw newfile .rw
<6>$ rwcat sorted .rw --output - path = new2 .rw \
--note - add =" originally from slowfile .rw , filtered for dport 22/ TCP <7>$ rwfileinfo new2 .rw --field = command -lines , annotations
new2 .rw: annotations
1 originally from slowfile .rw , filtered for dport 22/ TCP
4.1. MANIPULATING FLOW RECORD FILES 89
Example 4.7: rwfileinfo for Sets, Bags, and Prefix Maps
<1>$ rwfilter --type =out , outweb --start - date =2009/4/21 \ -- protocol =0- --pass = stdout \
| rwset --sip - file = internal . set
<2>$ rwfilter --type =out , outweb --start =2009/4/21 \ -- protocol =0- --pass = stdout \
| rwbag --sip - flow = internal . bag <3>$ rwfileinfo internal . set
internal . set :
format (id) FT_IPSET (0 x1d )
version 16
byte - order littleEndian
compression (id) none (0)
header - length 175 record - length 1 record - version 3 silk - version 3.5.1 count - records 3328 file - size 3503
ipset 16- way branch , root@3 , 33 x 80b nodes , 86 x 8b leaves
command - lines
1 rwfilter --type =out , outweb --start =2009/4/21 -- protocol =0- --pass = stdout
2 rwset --sip - file = internal . set <4>$ rwfileinfo internal . set -- fields = ipset
internal . set :
ipset 16- way branch , root@3 , 33 x 80b nodes , 86 x 8b leaves
<5>$ rwfileinfo internal . bag -- fields = bag internal . bag :
bag key : sIPv4 @ 4 octets ; counter : records @ 8 octets
<6>$ rwfileinfo internal . pmap -- fields = prefix - map internal . pmap :