Getting Data In

Custom Filters

didier_again
Explorer

I'm using the Unversal Forwarder to 'monitor' log files on the clients but I just can't index everything forwarded, there's too much data.

I know I can use REGEX to filter the data before it's indexed using the nullQueue example in the DOC. That's fine, it's working.

My problem is that filtering by REGEX is not flexible enough for what I need. Ideally I'd like to plug in a script (python or other) to only let some of the data reach the Indexer.

Is that possible at all? Or am I chasing a dead-end?

Regards,
Didier,

Tags (1)
0 Karma
1 Solution

Ayn
Legend

It is not possible to filter using anything other than regular expressions at index time.

One possible way to achieve this is to replace your file monitor input with a scripted input, and then implement all your filtering logic in the script you write for reading the input data.

View solution in original post

0 Karma

Ayn
Legend

It is not possible to filter using anything other than regular expressions at index time.

One possible way to achieve this is to replace your file monitor input with a scripted input, and then implement all your filtering logic in the script you write for reading the input data.

0 Karma

didier_again
Explorer

That should do. Thank you.

0 Karma

DUThibault
Contributor

(Late comment) Depending on what you're trying to do, you could also use SEDCMD in props.conf to throw away the parts of the events that you don't want indexed.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...