Getting Data In

parsing specific events in logs at heavy forwarder

funlearning321
New Member

Hello,

I am new to splunk and learning it . I am trying the parse the events with specific keyword will dropping the other events from the logs at heavy forwarder. For example, below are the sample logs .

2018-02-21T18:00:13.119575+00:00 apachefront audispd: node=abc.corp.com type=PATH msg=audit(1550772013.107:10434531): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=786685 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL 2018-02-21T18:00:13.1154665+00:00 apachefront audispd: node=apachefront type=EOE msg=audit(1550772013.107:10434531): 2018-02-21T18:00:13.120488+00:00 apachefront audispd: node=apachefront type=SYSCALL msg=audit(155054653.115:103534532): arch=c000003e syscall=59 success=yes exit=0 a0=1053420 a1=10534e0 a2=1050980 a3=7ffe6956c490 items=2 ppid=39078 pid=39084 auid=708926886 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=15253 comm="ps" exe="/bin/ps" key="root" 2018-02-21T18:00:13.12561541+00:00 apachefront audispd: node=apachefront type=EXECVE msg=audit(155564013.115:104455432): a0="ps" a1="-eT" 2018-02-21T18:00:13.121049+00:00 apachefront audispd: node=apachefront type=CWD msg=audit(16872013.115:1062): cwd="/" 2018-02-21T18:00:13.121241+00:00 apachefront audispd: node=apachefront type=PATH msg=audit(1550772013.115:10434532): name="/usr/bin/ps" inode=1995646 dev=fe:02 mode=0100755 ouid=0 rdev=00:00 nametype=NORMAL 2018-02-21T18:00:13.156434+00:00 apachefront audispd: node=apachefront type=PATH msg=audit(1550765463.115:10434532): item=1 name="/lproc/ld-linux-x86-32" inode=7865644 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL

In the logs, i am trying to make heavy forwarder to send the events that have type=SYSCALL and type= EXECEVE while dropping the others. Below is my transforms.conf, however heavy forwarder is dropping the all the events. Any help would be appreciated.

[set_SYSCALL]
REGEX =  \,\d{3}\s*\w+\s*\[type=SYSCALL]
DEST_KEY = queue
FORMAT = nullQueue

[set_EXECVE]
REGEX =  \,\d{3}\s*\w+\s*\[type=EXECVE]
DEST_KEY = queue
FORMAT = nullQueue

Thank you

0 Karma

woodcock
Esteemed Legend

Try this:
In props.conf:

[YourSourcetypeHere]
TRANSFORMS-set = setnull, setkeep

In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setkeep]
REGEX =  \,\d{3}\s*\w+\s*\[type=(?:SYSCALL|EXECVE)]
DEST_KEY = queue
FORMAT = indexQueue
0 Karma

pkeenan87
Communicator

It looks like you are sending both stanzas to the nullQueue which is why it is dropping all of the data. You will need to send only those groups to the TCP out queue and drop all of the other data by setting defaultgroup to nothing in outputs.conf

*** Props.conf
[your_sourcetype_here]
TRANSFORMS-syscall = set_SYSCALL
TRANSFORMS-execve = set_EXECVE

*** Transforms.conf
[set_SYSCALL]
REGEX = \,\d{3}\s*\w+\s*[type=SYSCALL]
DEST_KEY = _TCP_ROUTING
FORMAT = splunk

[set_EXECVE]
REGEX = \,\d{3}\s*\w+\s*[type=EXECVE]
DEST_KEY = _TCP_ROUTING
FORMAT = splunk

*** outputs.conf
[tcpout]
defaultGroup = nothing

[tcpout:splunk]
server= your_splunk_ip_here:9997

This documentation will be a good reference point for you on this: https://docs.splunk.com/Documentation/Splunk/7.2.4/Forwarding/Forwarddatatothird-partysystemsd

0 Karma

chrisyounger
SplunkTrust
SplunkTrust

Try this

[apachefront-setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[apachefront-setparsing]
REGEX = \stype=(?:SYSCALL|EXECVE)\s
DEST_KEY = queue
FORMAT = indexQueue

And in props.conf call it like so:

[sourcetype_goes_here]
TRANSFORMS-set= apachefront-setnull,apachefront-setparsing

Good luck

0 Karma

funlearning321
New Member

Thank you for the reply. i have tried the configuration that you provided and changes the regex to match only the type that i am looking for. However, splunk is still processing all the events.

REGEX = type=(SYSCALL|EXECVE)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...