Getting Data In

How to edit props.conf and transforms.conf to keep specific events and discard the rest?

tmontney
Builder

props.conf

[host::192.168.1.20:514]
TRANSFORMS-set= setnull,sra

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[sra]
REGEX = m=236
DEST_KEY = queue
FORMAT = indexQueue

Basically, I want my Splunk server to accept any event where field "m" equals 236 for a certain host. All other events get discarded. After putting both in "Splunk\etc\system\local" and restarting Splunk, I'm still getting all events.

Here's an example of _raw:

Jan 20 14:49:37 10.11.83.1 id=firewall sn=123456789ABC time="2015-04-17 19:48:01 UTC" fw=192.168.1.96 pri=6 c=16 m=236 msg="An error has occurred" sess="Web" n=11 usr="admin" src=192.168.1.200 dst=192.168.1.95 proto=tcp/800

woodcock
Esteemed Legend

Try this.

props.conf:

[host::192\.168\.1\.20]
TRANSFORMS-set= setnull,sra

transforms.conf"

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[sra]
REGEX = \s+m=236\s+msg=
DEST_KEY = queue
FORMAT = indexQueue

Put these files on every Indexer and restart all splunkd instances there. Events that arrive and are indexed after this will be correct; older events will stay as they were.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi tmontney,
if in your logs you have "m=236", you can filter your logs in the way you used, but correcting the regex:

 REGEX = m\=236

Beware that if you insert props.conf and transforms.conf in $SPLUNK_HOME/etc/system/local, you cannot manage them by Deployment Server and to deploy them in many servers will be very heavy!
Every way it's a best practice to manage configurations in Apps or Technology Add-Ons.

Bye.
Giuseppe

0 Karma

somesoni2
SplunkTrust
SplunkTrust

The REGEX attribute, during index-time operations, only works fields specified here, and by default applied to _raw. It doesn't apply to any custom field that you might have. If the literal string m=235 is not there in raw data, it won't be effective. You need to write a REGEX which will work on your raw data to filter/drop events that you want. If you can provide some sample events, of both you want to keep and drop, and highlight where the field m value is in there, we may suggest something.

https://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Transformsconf#KEYS:

0 Karma

tmontney
Builder

Also, updated to show an example of an event.

0 Karma

tmontney
Builder

You're saying because fields haven't been extracted at this point, yeah? The literal string, m=235, is in there.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

The REGEX should match exact format where that literal string is available, check if there are any extra spaces in between. Also, this should be setup on Indexers/Heavy forwarder.

0 Karma

tmontney
Builder

Never mind, I realized I specified the port number in the address.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

So does it work after your remove the port?

0 Karma

tmontney
Builder

The thing is, I'd expect to be getting no information. But I'm still getting all. Is my props file correct?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Is that the exact host name? If all the data that you're receiving is for single sourcetype, use that (is much simpler). Also, How are you getting the data, through Universal forwarder? If yes, then is UF sending data to Indexers or some heavy forwarder? The props/transforms should be placed there. Other than that (location of conf files and validity of stanza name in props.conf), your configurations looks good.

0 Karma

tmontney
Builder

I'm getting data from a network device via syslog. I cannot configure what it sends me. It's all or nothing.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

How about you try this for REGEX in transforms.conf

REGEX = \s+m=236\s+
0 Karma

tmontney
Builder

Does match exact format. It's one character field name equals a number.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...