Splunk Search

Negative Event searches

jsmith39
Path Finder

How would you search an application log for the absence of one or more specific events in a given time period?

I'm looking through event logs on multiple servers for specific records and am generating a report from them. So if an event of 12345 transpires, it's diplayed by this
sourcetype=applicationlogs tag=serverset eventcode=12345 | stats count by host

if it doesn't then that particular server just isn't listed in the report output. I'd like to note the absence and state "didn't happen".

Tags (1)
0 Karma

jsmith39
Path Finder

Thanks for the input everyone, I ended up going a more brute force method that seems to be working.
I tagged all of the servers to be searched, then filtered them by SourceName so I only had a working pool of logs that were related to the software application I'm looking at. It's a bit more verbose than I wanted but I'm hoping as I get more data I'll be able to refine it.

tag=Backup SourceName="Backup Exec" | eval Outcome=case(EventCode==57755 OR EventCode==34112, "Success", EventCode==34113 OR EventCode==34114, "Failure", NOT EventCode==57755 OR EventCode==34112 OR EventCode==34113 OR EventCode==34114, "Missed Jobs") | stats count by host, Outcome | rename host as "Server Name"

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try this:

sourcetype=applicationlogs tag=serverset 
| eval eventcode=if(match(eventcode, 34112), eventcode, "----") 
| stats count,values(eventcode) as eventcode  by host 
| eval isPresent=mvfind(eventcodes,"34112")
| eval count=if(isnull(isPresent),"Didn't happen", count)
| fields - isPresent, eventcode
0 Karma

jsmith39
Path Finder

theeven: I had to modify what you wrote to get it to run
tag=Backup| eval status=if(matcheventcode==34112, "yes", "no") | search status=no | stats count by host

Perhaps I modified it too much, but the end result is that I'm getting the book thrown at me as every record that isn't the matched eventcode is returned.

martin_mueller: Same scenario, I had to modify your query in order to run it, and perhaps I went too far but the results were not as expected. Strangely enough, there wasn't any statistics view and the records found were filtered, but I'm still trying to figure out in what way. The results are less than everything except 34112 but more than just noting an absence, for example I'm getting eventcode=0 and eventcode=58061 among other results.

tag=Backup | fields host | join type=left host [search tag=Backup eventcode=34112 | stats count by host] | fillnull count

Appreciate your help regardless

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Replacing the call to metadata with a search for some events does indeed change my search significantly, so that cannot work.You could replace metadata with this:

search for all events | dedup host | fields host

What didn't work for you?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Here's a thought (untested):

| metadata type=hosts index=yourindex | fields host | join type=left host [search index=yourindex blah blah eventcode=12345 | stats count by host] | fillnull count

Rows with count=0 are the droids you're looking for.

Note, I'm assuming all hosts in that index are potential senders of eventcode=12345. If that's not the case then you need to compute a list of all relevant hosts there instead of using metadata.

0 Karma

theeven
Explorer

If I understand correctly, here's what you looking for.

sourcetype=applicationlogs tag=serverset | 
eval status=if(match(eventcode, 12345), "yes", "no") | 
search status=no |
stats count by host
0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...