These could be real time searches.
I ran a search like "index=*" for 30 seconds realtime, and the apiStartTime was displayed as Zero_time
search total_run_time _time apiStartTime apiEndTime search_type user
search index=* 2018-03-20 10:28:09.913 ZERO_TIME ZERO_TIME ad hoc test_user01
search index=* 2018-03-20 10:28:13.560 ZERO_TIME ZERO_TIME ad hoc test_user01
The audit log captures the time range of the search. As a Splunk user, you specify the time range by using the pull-down menu (or by using the earliest
and latest
keywords). When Splunk processes the search, it calculates the actual time that should be searched. apiStartTime
represents the earliest time, and apiEndTime
represents the latest time.
EDIT - in my original answer, I said
apiStartTime='ZERO_TIME', apiEndTime='ZERO_TIME'
means that the search ran over All Time. It makes sense that this would be an excessively expensive search.
but this appears not to be the case.
END EDIT
No problem - please post if you figure it out...
Sorry but, indeed, it seems that your original answer is wrong.
A simpler search, without apiStartTime='ZERO_TIME' apiEndTime='ZERO_TIME', returns a bunch of other records, including the very same query, with the exact time range selected by the user. And this query occured just microseconds before the one with ZERO_TIME. So it must be something splunk does, but because it happens all the time it can't mean that it's the "All time" time range that was used.
So I have to remove the point. I will add this in a splunk ticket I opened to resolve cold storage searches that take our system down.
@sansay ,
Could you please let me know wht this actually means if you are aware of it now?
apiStartTime='ZERO_TIME', apiEndTime='ZERO_TIME'
Sorry but no, I haven't figured it out. I haven't had the time to even think about this issue.
Perhaps I am wrong. Could this have been something run by Splunk internally?
This gets weirder and weirder, according to my last search, and if apiStartTime='ZERO_TIME', apiEndTime='ZERO_TIME' means "All time", even I ran "All time" queries. This is starting to sound more and more like a bug.
Thank you very much lguinn.
The weird thing is that I disabled the "All time" from the GUI. And the user, from being the previous Splunk admin knows very well not to run "All time" queries. And he confirmed that when asked. So how else could this happen?
Is there any way I can get the exact query that was executed, ie, with the time range specified by the user?