Deployment Architecture

pftop parsing

danlynch
New Member

Hello,
I've added a unix script to that identifies all NATed traffic across my BSD firewall (pf):

/usr/local/sbin/pftop -ab -v long -w 140 

The output is in this format:

PR    DIR SRC  DEST   GW  STATE  AGE       EXP  PKTS BYTES   AVG RU

in props.conf I have:

[pftop]
SHOULD_LINEMERGE = false
LINE_BREAKER = ^()$
TRUNCATE = 1000000
DATETIME_CONFIG = CURRENT
REPORT-pftop_fields = pftop_fields

in transforms.conf:

[pftop_fields]
REGEX = (tcp|udp) (in|out) (\d+\.\d+\.\d+\.\d+)\:?(\d*) (\d+\.\d+\.\d+\.\d+)\:?(\d*) (\d+\.\d+\.\d+\.\d+)\:?(\d*)   (\w+) (\d*) (\d*) (\d*) (\d*) (\d*) (\d*)
FORMAT = proto::$1 direction::$2 src_ip::$3 src_port::$4 dest_ip::$5 dest_port::$6 gw_ip::$7 gw_port::$8 state::$9 age::$10 expires::$11 packets::$12 bytes::$13 ave::$14 rule_number::$15
CLEAN_KEYS = 1
MV_ADD = 0

My problem is that splunk is parsing the output so each time the script runs it is seen as one event. Not only would I like to be able to see individual connects but also search on some of the key variables. netstat is experiencing the same difficulties on my system. Any help would be appreciated.

Update 2 Mar 13:
I reformated the the stdout, but I'm not sure how you would add a timestamp to each line. Below is the code for the shell script:

. `dirname $0`/common.sh
HEADER='PR     DIR    SRC                     DEST                   GW                   STATE                   AGE         EXP          PK
HEADERIZE="BEGIN {print \"$HEADER\"}"
PRINTF='{printf "%-5s  %-5s  %-21s  %-21s  %-21s %-23s %-10s  %-10s  %6s  %6s  %5s  %3s\n", $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12}'
CMD='eval /usr/local/sbin/pftop -ab -v long -w 140 | grep "tcp\|udp\|icmp"'
FORMAT='{gsub("[46]","",$1); if(!$12) { for (j=13; j>5; j--) $j=$(j-1); $5="N/A"} }'
assertHaveCommand $CMD
$CMD | tee $TEE_DEST | $AWK "$HEADERIZE $FIGURE_SECTION $FILTER $FORMAT $FILL_BLANKS $PRINTF"  header="$HEADER"
echo "Cmd = [$CMD];  | $AWK '$HEADERIZE $FIGURE_SECTION $FILTER $FORMAT $FILL_BLANKS $PRINTF' header=\"$HEADER\"" >> $TEE_DEST

Sample output:

PR  DIR SRC             DEST           GW  STATE                   AGE        EXP      PKTS BYTES AVG RU
tcp Out 127.0.0.1:14801 127.0.0.1:7736 N/A ESTABLISHED:ESTABLISHED 503:29:07  00:05:58 42K    200M  115 1
Tags (1)
0 Karma

Ayn
Legend

Turn off line merging.

In props.conf:

SHOULD_LINEMERGE = false
0 Karma

danlynch
New Member

It is set to false but it doesn't seem to help

0 Karma

Kate_Lawrence-G
Contributor

I've found the easiest away around this to reformat the stdout of the info to create an "event" with strict field assignments and a time data stamp so that when Splunk reads it it will be indexed by the time format I set in the props.conf.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...