I got a different result count when I executed this query a week before, and when I executed it today. The first time, the query returned 16 records, today, it returned 21! How is this possible? I ran the search for the same absolute time period both times. If it helps, I experienced similar inconsistent results with another query on the same search head. There are no errors in the search results that could point to any suppressed events:
servername=abc* sourcetype=bq
| rex "java\.\S+\.(?P<Var1>[ A-Z]+(Err))"
| rex field=_raw "(?<Var2>com\.jss\S*\.\S+)\.[A-Z]\S+\((?<Var3>\w+)\.java:(?<Var4>\d+)\)"
| search Var1=NNN
| eval Var3=coalesce(Var3, "No Var3"), Var4=coalesce(Var4, "No Var4"), Var3=Var3. "." .Var4
| search Var3=*
| stats count by Var1, Var3, Var2
I have already spent many hours trying to troubleshoot this, so any pointers would be very helpful. Thank you!
This is 1 of 2 problems:
1: The events are arriving late. Sometimes the box is completely off, or offline, or Splunk is not running and then it comes back and the events come flooding in late.
2: If you are using an accelerated datamodel, this usually runs behind about 3 minutes but sometimes WAY more than that, especially if you rest it.
You can compare _indextime
against _time
to differentiate between the 2.
Thank you for your reply, @woodcock, would you mind sharing how I could compare _indextime
against _time
? Do I remove the stats
statement at the end of my query and simply append _indextime
and _time
to the search
statement? Thanks!
Download the Meta Woot!
app and it will make it easy to see.
Thanks, can this be installed by someone who's not a Splunk Admin? From a quick online check, it seemed not, but maybe I'm mistaken.
Tagging @somesoni2, @woodcock as they have been very helpful with such questions before and this is a little urgent. Thank you
P.S. - This is on Splunk Enterprise v7.1.6