Splunk Search

Why enabling real time search does not display data immediately?

asingla
Communicator

I am receiving events every 15 seconds. But when I enable real time search in default splunk search app for query sourcetype="mysource" with 1 minute window, It does not display the events arriving after enabling this search. If I see the timeline, it shows the current time is proceeding but no event appears. I see the the count against the scanned events but matching events remains 0. It takes a while (almost 3-4 minutes) after which the events starts appearing as expected.

If I enable real-time for 'All Time' then everything works as expected.

Is there anything which I am missing about the real-time search.

Tags (2)
0 Karma
1 Solution

jflomenberg
Splunk Employee
Splunk Employee

It sounds like 1 of 2 things may be happening.

  1. Events could be arriving late. If events take more than 60s to reach the indexer then they wont meet the time bounds of rt-1m to rt (1 minute window) and won't be displayed. If you are receiving events every 15s then this seems less likely.
  2. You have some clock skew going on. If events have older time stamps they effectively reduce the 60s window. If all events are time stamped 1 minute in the past then the window is effectively zero.

Based on the comment about waiting 3-4 minutes for an event, it sounds like your indexer may be ahead 4 minutes relative to the data source.

View solution in original post

jflomenberg
Splunk Employee
Splunk Employee

It sounds like 1 of 2 things may be happening.

  1. Events could be arriving late. If events take more than 60s to reach the indexer then they wont meet the time bounds of rt-1m to rt (1 minute window) and won't be displayed. If you are receiving events every 15s then this seems less likely.
  2. You have some clock skew going on. If events have older time stamps they effectively reduce the 60s window. If all events are time stamped 1 minute in the past then the window is effectively zero.

Based on the comment about waiting 3-4 minutes for an event, it sounds like your indexer may be ahead 4 minutes relative to the data source.

asingla
Communicator

Thanks. It was the second reason.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...