Filter Log Data on Matches

[kwl33181]

Is it possible to filter log data by matching values when filtering with a heavy forwarder. I have multiple universal forwarders sending modsecurity, and apache error logs to a heavy forwarder and then onto an indexer. Is it possible to filter the apache access logs at the heavy forwarder so that only the access log events that match and error or modsecurity log entry would be forwarded on to the indexer. Sadly The log volume of the access logs them self are too large to forward and index so I need a way to filter to a subset of useful information.

[yongly]

If you want to just filter the access logs from your apache server, then you can do so by filtering on a term or regular expression just like jbsplunk has specified.

I have a similar set up with a webserver sending ssl_access_log to an intermediary heavy forwarder which then filters and only forwards events with a string of /WPNotification to the indexer.

Not sure what other kind of filtering you might be wanting to do??

[jbsplunk · Accepted solution]

Yes, it is possible. You can route data to what is called the 'nullQueue'

http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad

start with this:

You can eliminate unwanted data by routing it to nullQueue, Splunk's /dev/null equivalent. When you filter out data in this way, the filtered data is not forwarded or added to the Splunk index at all, and doesn't count toward your indexing volume.

[jbsplunk]

No, you can't make the regex based on some already indexed data. I don't think you can selectively filter in the manner you are describing.

[kwl33181]

Correct, but can you make a regex based on other incoming data? Thought process is when a modsecurity event trips for value X then I want the accesslogs associated with X. But X changes dynamically based on the various errorlog or modsecurity log events. I think you have me going the right direction, but I am missing the triggering or matching(to dynamic value) piece.

[jbsplunk]

As long as you can come up with a regex to match on, you should be able to use this method. If you want to tell Splunk to only look at particular files, you can use blacklist/whitelist in the input stanza.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Whitelistorblacklistspecificincomingdata

[kwl33181]

This is great for limiting what gets forwarded, but how do I use the errorlog or modseclog events as the matching parameter before null routing? Even a method to forward a subset of events based on uri or time stamp would be useful.