Understanding forwarding, filtering and license consumption

[wemb]

Hello. I've inherited a 'proof-of-concept' Splunk installation consisting of several linux servers running Splunk Enterprise under a dev license. We've a couple of Indexers, an index master, a deployment server and a single search head.

We've got universal forwarders configured on our Windows AD domain controllers that are installing the Spunk_TA_Windows app to the UFs. This app has been configured to whitelist only certain event ID codes via a regexp.

My understanding (and this appears to agree with every bit of docs I can find) was that UF forwarded were unable to filter or manipulate data, and that required a Heavy Forwarder to be configured to do things like Regexp,etc.

I'm concerned that our UF are consuming too much of our license and that the whitelisting Regexp's in the inputs.conf on the UF aren't effective? Or have I grossly misunderstood (I assume it's me...)

The other reason I ask is because we'd also like to pull in selected data from a syslog feed, but we'd absolutely need to filter this before it hits splunk as it'd blow through our license in mins if we didn't. If I can filter windows event logs in a UF via a regexp - can I also filter syslog events in a UF with a regexp ?

Thanks
Dave

[gcusello · Accepted solution]

Hi wemb,
if you filter events before indexing, the filtered events don't consume license.
Filtering can be applied on Indexers or (when present, but it isn't a need) on Heavy Forwarders, so if you want to filter events you can do on Indexers without an additional HF.
For additional information see https://docs.splunk.com/Documentation/Splunk/7.3.2/Forwarding/Routeandfilterdatad

Universal Forwarders don't filter data with the only exception of Windows eventlogs that it's possible to filter events on Universal Forwarder, whitelisting or blacklisting EvenCodes.
So, if your UF are consuming too much of your license and that the whitelisting Regexp's don't run, probably this is the pèroblem.
Then if you need to filter syslogs, you can do easily on Indexers (see the above documentation) not on UFs.

You can receive syslogs on Indexers but (when possible) I prefer to use two Heavy Forwarders with a Load Balancer to ingest syslogs, in this way I can ingest syslogs without overload Indexers, I can filter them and I can separate this job from the Indexers' jobs (you need two HFs and a Load Balancer to avoid Single Points of Failure).

If you use the Windows TAs, you have to enable only the events you need for your use cases: e.g. you could want only security WinEventLogs but not performance monitoring logs, i addition for scripts, you could also change the frequency of execution (e.g. hardware specs only one time at a dayand not everyten minutes.

Ciao.
Giuseppe

[wemb]

Thanks Giuseppe - it's the "UF's don't filter data with the exception of Windows eventlogs" that was news to me. That explains why we don't have any HF, but our events are still being filtered.

I'll try setting up a heavy Forwarder to filter our syslog data before sending it on to Splunk.

Thanks
Dave

[gcusello]

Hi wemb,
I suggest to use two HFs not for filtering, this is an additional feature, but mainly to manage ingestion in a separate way then indexers.
If you want to use an HF only to filter data, you don't need it!

Ciao.
Giuseppe

[wemb]

Sorry - I'm confused again now - I'm talking about syslog data now, not windows Event logs - can I filter syslog data to avoid consuming my license on a UF?

[gcusello]

No, you can filter your syslog events on Indexers (before indexing) or Heavy Forwarders not on Universal Forwarders!
Filtering on Indexers doesn't consume license because filtering is an action before indexing and license is calculated only on indexed logs not on received logs.

About Heavy Forwarders for syslogs, if you have a small architecture, you can use Indexers to receive syslogs and filter them before indexing.
If instead you have a large architecture and you want to separate syslog receiving from indexing you can add one (two with a Load Balancer is better) Heavy Forwarder (not Universal Forwarder!) that it's enabled to receive syslogs and filter them before sending to indexers.

Ciao.
Giuseppe