Getting Data In

What is the frequency with which logs are read in Splunk?

garimayadav
New Member

What is the frequency with which logs are read in Splunk? Does delay in seeing recent log details in Splunk related to poor indexing?
And if indexing is perfectly fine, then what should be the ideal time after which details of a new request must be visible in Splunk?

Tags (2)
0 Karma

satishsdange
Builder
0 Karma

MuS
SplunkTrust
SplunkTrust

Hi garimayadav,

an universal forwarder will poll / check the files once a second.

cheers, MuS

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

From my understanding, there is no "frequency." If the file changes, it starts forwarding the new data. There are technically several ways to do this, I don't know which one Splunk uses, but I have used a few throughout my programming career.

How soon should you see indexed data in Splunk? That depends on the processing of the data at index time. If not much is being looked at, you can have data visible sub-second after it is placed in a file on a forwarder, if all the stars align properly. We index about 350GB/day and see most of our data within 10 seconds with most of that under 5 seconds. I have had "Last 5 seconds" searches that fill up even the last 1 second slot with almost as many events as the other 4 slots.

If your indexing is putting structure on the data, or the indexing has to search hard to find the timestamp, or your network is slow, or the indexing hardware is slow (especially the disks), these things can lead to slow indexing of the data.

0 Karma

garimayadav
New Member

Thanks Cary 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...