Splunk Search

Dissimilar results between sid & normal search

Naren26
Path Finder

I am facing a weird issue with sid. I have a saved sid with yesterday's (00:00 to 23:59) data, which is showing a dip in my messages during yesterday evening. But, for the same query (since there is no change in the query, i believe I don't need to provide my query here) and for the same time frame, if I try to run as a new search, I do not see the dip.

Since, both are having same query & timeframe, how the results would change? Is there any difference in this? Someone please explain.

0 Karma

woodcock
Esteemed Legend

Like @daljeanis implied, the problem is probably latency in your event delivery pipeline such that when you search for events way later, you see more of them because you gave them all enough time to get indexed and searchable. You can check latency either with the Meta W00t app or with a search like this:

| tstats max(_indextime) AS _indextime WHERE index=YourIndexHere AND sourcetype=YourSourcetypeHere BY _time host
| eval maxLatencySeconds = (_indextime - _time)
| stats avg(maxLatencySeconds) BY host
0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Try adding this to your later search

| where _indextime <= relative_time(now(),"@d")

and see if they go away.

If you know the exact time of your prior search, you could substitute an epoch-format version of that in the test.

0 Karma

woodcock
Esteemed Legend

You are missing something there, @daljeanis.

0 Karma

janispelss
Path Finder

Could it be that some of the data in that time range was indexed after you completed your original search, the one with the sid you're using?

Naren26
Path Finder

It seems so!! Whatever I modify in that sid, there is no dip. May be caused by delay indexing??

0 Karma

janispelss
Path Finder

That does seem the most likely cause. The sid returns the exact results from when the search was originally run, apparently with incomplete data. Any new searches however have all the data available.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...