Splunk Search

Why is my search for missing events returning results from clients that actually do have events?

colbymahan
Explorer
SourceName="EBS Check" OR SourceName="EBS Snapshot" | eval hasEBSCheck=1 | append [| metadata type="hosts" | eval hasEBSCheck=0] | stats max(hasEBSCheck) as hasEBSCheck by host | search hasEBSCheck=0 | search host=*-SLX NOT host=*-STG NOT host=*-SANDBOX NOT host=demo-slx 

This search is returning 3 clients as having 0 events in the past day from those SourceNames, but they do in fact have the events. Only 3 of 250+ are coming back incorrectly. What could be reasons that these 3 are incorrectly calculated? Clicking on "Events" shows that in fact they have them and the count should not be "0".

0 Karma

colbymahan
Explorer

OK I think I know why now, but not how to fix it. Those hosts have events with differing case on the hostname from previous servers that were upgraded. It looks like splunk is seeing aic-slx as a different server than AIC-SLX. Since there are no recent events for the older name, it is returning that as a result. How can I get the report to see aic-slx and AIC-SLX as the same server? Adding NOT AIC-SLX causes the host to not bee included at all for both cases since Search is case insensitive. It seems like the lookup/meta data is the part that is differentiating case to see 2 different servers.

0 Karma

lguinn2
Legend

I rewrote the search slightly; I think this makes it more efficient, but I don't believe that it makes the search more accurate.

SourceName="EBS Check" OR SourceName="EBS Snapshot" host=*-SLX NOT host=demo-slx 
| eval hasEBSCheck=1 
| stats first(hasEBSCheck) as hasEBSCheck by host
| append  [ metadata type="hosts" 
       | search host=*-SLX NOT host=demo-slx 
       | eval hasEBSCheck=0 ] 
| stats max(hasEBSCheck) as hasEBSCheck by host 
| search hasEBSCheck=0 

To see if anything is going wrong with a search (either this one or yours), you can take a look at the Search Job Inspector and the search.log. Run the search, and then click the Job> link underneath the search bar to find the Job Inspector to examine how the search executed. The search job inspector also has a link to the search.log at the bottom of the page.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Give this a try

SourceName="EBS Check" OR SourceName="EBS Snapshot" | stats count as hasEBSCheck by host | append [| metadata type="hosts" | table host | eval hasEBSCheck=0] | stats max(hasEBSCheck) as hasEBSCheck by host | search hasEBSCheck=0 | search host=*-SLX NOT host=*-STG NOT host=*-SANDBOX NOT host=demo-slx
0 Karma

colbymahan
Explorer

While this "works" like the other search, it still is returning 3 hosts that show 0 in the chart but when you look, the events are there. So same results... Thanks for the suggestion though...

0 Karma

lguinn2
Legend

I like the "stats count as hasEBSCheck"
Very efficient

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...