Why am I unable to filter out an event code?

[marcmuher]

I apologize since this has been answered before. I've tried all the solutions offered in the previous threads and am still not able to get it working.

I need to filter out event code 5156 coming into our Splunk instance. We're seeing about 25,000 of these events per minute (possibly a Kaspersky issue on the servers coming in over universal forwarders, but I can't do anything about that) and I'm trying to lessen our total usage.

We're on Windows, using Splunk Enterprise 6.5.1. I've tried editing the inputs.conf file in splunk\etc\system\local and adding this entry:

[WinEventLog://Security]
disabled = 0
blacklist = "5156"

I've done it with and without the // in WinEventLog and with and without the "" around 5156, but am still seeing the same amount of 5156 events coming through in Splunk as before. And I'm obviously pretty new with Splunk and my position in this company so I'm trying to get something working correctly

[marcmuher · Accepted solution]

Thank you for the replies. According to our Splunk after-sales engineer, we need to change the input.conf on the systems that have a universal forwarder, not just the heavy forwarder. Once I get our network guys to make these changes, I'll post whether it worked or not.

[marcmuher]

Indeed, changing the inputs.conf on the systems with the universal forwarder was the solution when using 6.5.1. I changed the biggest offender this morning, and it looks like it will reduce our data consumption per domain controller by about 80%. For some reason, we were logging and then processing through Splunk every time a computer did a DNS lookup, which was the bulk of the 5156 codes.

I also learned something important. I had cut and pasted the

[WinEventLog://Security]
disabled = 0
blacklist = EventCode = ”5156”

from an email from our Splunk after-sales engineer into Notepad on the target computer. On the email the quotation marks are curved. Splunk does not appear to like curved quotation marks, so Splunk ignored the command and it did not work. Once I typed the quotation marks in manually ("5156"), it worked.

[bmo017]

Hello,

As @richgalloway stated, this change should happen at the forwarder level, that is if you are using a Heavy Forwarder, if not then this will have to happen at the indexing layer of the pipeline.

The format of your blacklist should be as below:

blacklist = EventCode = 5156

Please reference the doc below for further information:

http://docs.splunk.com/Documentation/Splunk/6.2.5/admin/inputsconf

[richgalloway]

Are you changing inputs.conf on the forwarder? Are you restarting the forwarder after each change?
Run btool to verify the settings are as you expect.
The blacklist attribute should not have quotation marks.

[marcmuher]

Thanks for replying so quickly!

It's the inputs.conf on the Splunk server itself - we don't have a heavy forwarder, just universal forwarders, so I didn't think I need to do anything to the servers with the forwarders on them.

I'm restarting the Splunk service whenever I make a change, although I am getting an error message when I stop the splunkd service that says "Windows could not stop the Splunkd service on Local Computer. Error 1503: The service did not respond to the start or control request in a timely fashion." But then it restarts normally.

I tried btool, but get the message "Splunk_home must be set. Stopping." and I only have the brain power to troubleshoot one problem at a time!

[richgalloway]

The admin manual says this about WinEventLog: "Note: The WinEventLog stanza is for local systems ONLY." That means the inputs.conf changes need to be made on the forwarders. You can also filter the events using props.conf on the indexer(s), but preventing the forwarders from sending the events will save network traffic.

You can add %SPLUNK_HOME% as a system-wide environment variable by using the Advanced tab in the System Properties dialog box.

[hgrow]

Hi there,

hi just can aid richgalloways idear of the inputs.conf you use is not in place.

The data you are talking about comes from a server wich is not your splunk server itself, since you sad the data is coming in from your forwarder. Even though your inputs.conf is written on the splunk-instance, it is deployed to your universal forwarders (UF) and needs to be active there. When you restart the splunk server, is it the splunk server or the universal forwarder after(!) it pulled the new configuration?

How are you deploying your inputs?. If you edit the inputs.conf in a deployment app ($SPLUNK_HOME$/etc/deployment-apps//local/inputs.conf) you can configure the app to autmaticly restart the Universal Forwarder after the app got redeployed. Everything to read should be found here:

http://docs.splunk.com/Documentation/Splunk/6.5.1/Updating/Createdeploymentapps
https://docs.splunk.com/Documentation/Splunk/6.5.1/Updating/Updateconfigurations

Besides a correct deployment process I've noticed something while going through the documentation about WindowsEvent-Monitoring. There is difference between the the two blacklist-filtering formats you can apply in your configuration. I'm pretty sure I've already used the 2nd "advanced filtering" mode already in some configurations. You can give it a try!

When using the Event Log code/ID format:

For multiple codes/IDs, separate the list with commas.
For ranges, use hyphens (for example "0-1000,5000-1000").

When using the advanced filtering format:

Use '=' between the key and the regular expression that represents your filter (for example "blacklist = EventCode=%^1([8-9])$%"
You can have multiple key/regular expression sets in a single advanced filtering entry. Splunk software conjuncts the sets logically. This means that the entry is valid only if all of the sets in the entry are true.
You can specify up to 10 blacklists per stanza by adding a number to the end of the blacklist attribute, for example blacklist1...blacklist9.

http://docs.splunk.com/Documentation/Splunk/6.5.1/Data/MonitorWindowseventlogdata

Greetings