Splunk Search

Same query run multiple times returns different results

rey123
Path Finder

I got a different result count when I executed this query a week before, and when I executed it today. The first time, the query returned 16 records, today, it returned 21! How is this possible? I ran the search for the same absolute time period both times. If it helps, I experienced similar inconsistent results with another query on the same search head. There are no errors in the search results that could point to any suppressed events:

servername=abc* sourcetype=bq
| rex "java\.\S+\.(?P<Var1>[ A-Z]+(Err))"
| rex field=_raw "(?<Var2>com\.jss\S*\.\S+)\.[A-Z]\S+\((?<Var3>\w+)\.java:(?<Var4>\d+)\)"
| search Var1=NNN
| eval Var3=coalesce(Var3, "No Var3"), Var4=coalesce(Var4, "No Var4"), Var3=Var3. "." .Var4
| search Var3=*
| stats count by Var1, Var3, Var2

I have already spent many hours trying to troubleshoot this, so any pointers would be very helpful. Thank you!

0 Karma

woodcock
Esteemed Legend

This is 1 of 2 problems:
1: The events are arriving late. Sometimes the box is completely off, or offline, or Splunk is not running and then it comes back and the events come flooding in late.
2: If you are using an accelerated datamodel, this usually runs behind about 3 minutes but sometimes WAY more than that, especially if you rest it.
You can compare _indextime against _time to differentiate between the 2.

0 Karma

rey123
Path Finder

Thank you for your reply, @woodcock, would you mind sharing how I could compare _indextime against _time? Do I remove the stats statement at the end of my query and simply append _indextime and _time to the search statement? Thanks!

0 Karma

woodcock
Esteemed Legend

Download the Meta Woot! app and it will make it easy to see.

0 Karma

rey123
Path Finder

Thanks, can this be installed by someone who's not a Splunk Admin? From a quick online check, it seemed not, but maybe I'm mistaken.

0 Karma

rey123
Path Finder

Tagging @somesoni2, @woodcock as they have been very helpful with such questions before and this is a little urgent. Thank you

P.S. - This is on Splunk Enterprise v7.1.6

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...