Monitoring Splunk

What is the frequency with which log files are read?

brettski
Explorer

I am trying to determine what the frequency is that Splunk reads log files. I have Data Inputs setup against 5 web servers in a farm reading W3C logs. It seems that it can take hours for Splunk to get to some of the logs. While one server is read pretty much instantly. I don't understand why there is a difference.

The log files are created hourly and rotated on a ~monthly basis. When I originally setup the Inputs I didn't specify to tail the file, I wanted all of the history with the initial load. It's been 2.5 hours and I still only have one server's data from the thirty minute time span I am looking at.

I guess my question is, does Splunk have the ability to tail multiple files at once? Is there a way to configure Splunk so I have more up to date log files index across my web farms?

UPDATE:
It has been five hours and I still only have the one server's logs. These Inputs have been setup for more than a month, it is not a case of an initial load delay. I have verified the logs are all in GMT, and they appear to be setup the same on the Data input (files) screen.

Can more than one file be tailed at a time?

Tags (2)

jbsplunk
Splunk Employee
Splunk Employee

It could be that Splunk just hasn't got through the volume of data that your asking it to go through. Do you have an idea of how much data actually exists within these folders in terms of size and number of files?

You may be able to get some additional information from the file input status monitor:

https://splunkserver:8089/services/admin/inputstatus/TailingProcessor:FileStatus

It may give you an idea of what has been read so far and help you determine how things are progressing.

Splunk does have the capability to tail multiple files at one time, but there are limits in that it does take time to process data.

I am not sure what your input configuration looks like. Could you perhaps paste it?

Sometimes it is helpful to do an Real-Time(All Time) search on the hosts to see which data is coming into Splunk real time via the time selector. The first thing I'll do when I don't see the data I think I should be seeing is an all time, real time search for the source/sourcetype/host.

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

It depends on the configuration, but yes, Splunk does start over again on restart to determine what files have been read and the last position read from those files.

0 Karma

brettski
Explorer

Is the Input status rebuilt each time the server is restarted? I have noticed that after a restart, the file count starts at one again.

0 Karma

brettski
Explorer

Sorry for the crap tags, my karma is too low to create tags more fitting to the question.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...