How to exclude result based on static exception file

[glitchcowboy]

I've got a script that checks various settings on every host and returns data to the indexer via universal forwarder. The output looks like this:

spec=1.2.3 result=pass remediation=
spec=1.2.4 result=fail remediation=fix things
spec=1.2.5 result=pass remediation=
(((host=x)))

spec=1.2.3 result=fail remediation=
spec=1.2.4 result=fail remediation=fix things
spec=1.2.5 result=pass remediation=
(((host=y)))

My exception file looks like this (but can look like anything I want it to)

host=x spec=1.2.4
host=y spec=1.2.3

My desired report would show this:

host x problems:
NONE!
host y problems:
spec 1.2.4

Note that host x spec 1.2.4 and host y spec 1.2.3 are in the exclude file and thus not reported.

Any thoughts on how I can pull this off in splunk?

[MHibbin]

On the assumption that you have already indexed your data... My first thought would be to use a lookup file in the form of csv (doc ref here). An example of your exceptions could be.

host,specExc
x,1.2.4
y,1.2.3

(you can have multiple "specExc" fields to check against in Splunk).

And then create your search and use the lookup command, to add the extra "specExc" field to your data (based on the "host" field). You will then need to use something like the eval command to filter out the results.

UPDATE: Given an example as requested...

So I loaded in the following data as events (I modified the host to be inline, to make it easier for Splunking, as you said things can be changed, this should be to hard, you should include the time/date in the events as well)

host=x spec=1.2.3 result=pass remediation=
host=x spec=1.2.4 result=fail remediation=fix things
host=x spec=1.2.5 result=pass remediation=


host=y spec=1.2.3 result=fail remediation=
host=y spec=1.2.4 result=fail remediation=fix things
host=y spec=1.2.5 result=pass remediation=

host=a spec=1.2.3 result=fail remediation=fix things
host=a spec=1.2.4 result=pass remediation=
host=a spec=1.2.5 result=pass remediation=

host=b spec=1.2.3 result=pass remediation=
host=b spec=1.2.4 result=pass remediation=
host=b spec=1.2.5 result=fail remediation=fix things

I had the events broken on each instance of "host="... If I was you I would add the date in the events as this is a good practise and makes it easier to look back through logs more accurately. I have assigned this to the sourcetype, "testestest".

So now I have each event as one block of results (per host), I would set up the lookup.. this would involve creating your csv file as previously describe in the "lookups" directory of an app (e.g. for the "Search" app it would be $SPLUNK_HOME/etc/apps/search/lookups, swap out "search" for your app where appropriate).

specExceptions.csv:

host,specExc
x,1.2.4
y,1.2.3
a,1.2.3
b,1.2.3

You then need to tell Splunk about the lookup file, so I normally edit the "transforms.conf" file in the same App's local directory, you may need to create one if not present (i.e. "$SPLUNK_HOME/etc/apps/search/local/transforms.conf"). This should look something like the following (where the stanza is the lookup name you want to reference in Splunk, and the filename is your CSV file)...

transforms.conf:

[testExeptions]
filename = specExceptions.csv

You can then check this in Splunk by using the "inputlookup" command, such as (note that the pipe is the first thing used, it's a special search command):

|inputlookup testExceptions

You should then see your CSV file. Now we can start getting creative with Splunk... First we collect our events, by using the search (remember the sourcetype I used)...

sourcetype=testest

We can then add the extra field from the lookup to the events based on the host. I have used the rex command to extract the host field (as "host" is metadata, it should be extracted using the conf files)...

sourcetype=testestest | rex field=_raw "host=(?P<host>\w+)" | lookup testExceptions host OUTPUT specExc

You should now see the extra field available in the "field discovery" panel. You can then do something like, using eval..

sourcetype=testestest | rex field=_raw "host=(?P<host>\w+)" | lookup testExceptions host OUTPUT specExc | table host spec specExc result remediation | eval action=if(result=="fail",case(spec!=specExc,remediation,spec==specExc,"nothing"),"nothing") | fields host, spec, specExc, result, action

To generate a table where it shows the host, spec, specExc, result, and action, where "action" would be populated with the "remediation" based on the lookup file (you probably won't need the fields listed, just thought I'd leave them in to show.

Hope this helps get you on your way.

Regards,

MHibbin

[MHibbin]

I have edited the answer above... I assumed you could modify your output slightly, to include host inline (as this would be a more descriptive/tidier manner IMHO) and you should also think about including date of each test inline as well, Splunk likes dates :-)...

Hopefully this makes sense, been a long day (-.-)zz

[glitchcowboy]

I'm developing this, so I have some indexed data, but I trash it at will. So if another solution is easier, I can start over.
Can you give me more specifics on how I would use the lookup?