Getting Data In

troubleshooting filtering at Heavy Forwarder with Props.conf / Transform.conf

k2skaterii
Path Finder

I am currently passing all logs through a Heavy Forwarder so I can filter out "noisy" logs before they are indexed. I am successfully filtering 4 other items including Windows Logs, SYSLOG, and Windows WMI data. Now I am trying to filter IIS logs, and am running into trouble.

I believe my props.conf and transforms.conf are setup correctly. Can someone help me figure out where this filter is failing?

Props.conf:
[iis]
TRANSFORMS-null2=IIS_Remove_F5_Health_Checks

transforms.conf:
[IIS_Remove_F5_Health_Checks]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueu

I realize my REGEX is going to filter EVERYTHING. I had a more specific REGEX in there previously, but I replaced it with the " REGEX = ." as part of my troubleshooting process. The original REGEX was supposed to filter out any logs with a 10.1.100.8 or 10.1.100.9 IP address, that REGEX was:

REGEX=(?<!\d)10.(?<!\d)1.(?<!\d)100.(?<!\d)(8|9)

The sourcetype for my ISS logs is "iis" (all lower case). In my mind it looks like all iis logs should be passing processed by the transform, but it looks like that is not the case. How can I troubleshoot this?

Thanks for the help!

dwaddle
SplunkTrust
SplunkTrust

Are your IIS logs at the forwarder using INDEXED_EXTRACTIONS? If so, data arrives at the heavy "cooked" already. Configure the forwarder to nullQueue it. Seems wrong, I know. But..

k2skaterii
Path Finder

The host that is forwarding the iis logs is using the default props.conf, which includes:

INDEXED_EXTRACTIONS = w3c

I do want INDEXED_EXTRACTIONS TO be active as that helps the logs be "Useful". If I disabled INDEXED_EXTRACTIONS on each of my host would the Heavy Forwarder be able to provide that function?

0 Karma

dshpritz
SplunkTrust
SplunkTrust

Yes, if you are not using indexed extractions, then the data is not sent cooked, so the HF will be able to filter.

dwaddle
SplunkTrust
SplunkTrust

Or, configure your nullQueue entries for [iis] directly on the UFs themselves. They will filter BEFORE things get to the heavy.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...