Splunk Search

How to filter logs from the source with a universal forwarder?

Log_wrangler
Builder

Hi,

I have UFs on a few ec2 aws instances, reading logs from /temp.

I want to regex and only send logs containing ERROR and WARN on to the HF and then on to the indexers.

I want to the filter to occur closest to the source to reduce the amount of data being sent.

Is it possible to regex in the inputs.conf of the UF? If so please explain.

Thank you

0 Karma
1 Solution

xpac
SplunkTrust
SplunkTrust

Hey, the UF has no real concept of lines, or events, it just see's blocks of data and sends that to the indexers for processing.
It's therefore not possible to regex-filter on the inputs (with a few rare exemptions, for example on Windows event logs).
You'll either have to built a scripted input that reads the files and filters them for you, or install an HF, or just send the logs and have the first HF/indexer they reach do the filtering.

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

View solution in original post

FrankVl
Ultra Champion

What is generating those logs? Any chance you can have that write error/warn events to a separate file?

0 Karma

xpac
SplunkTrust
SplunkTrust

Hey, the UF has no real concept of lines, or events, it just see's blocks of data and sends that to the indexers for processing.
It's therefore not possible to regex-filter on the inputs (with a few rare exemptions, for example on Windows event logs).
You'll either have to built a scripted input that reads the files and filters them for you, or install an HF, or just send the logs and have the first HF/indexer they reach do the filtering.

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

Log_wrangler
Builder

Thanks for confirming, I was pretty sure I would need to use the HF to regex and send, just wanted to make sure.

0 Karma

xpac
SplunkTrust
SplunkTrust

Be aware that data pre-processed by an HF is much larger on the network than what an UF sends. You might add complexity without reducing the data stream, plus you have to do all the parsing work on the HF then 🙂

0 Karma

Log_wrangler
Builder

yes that is a good point.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...