Getting Data In

file size impact continuous monitoring in splunk?

rameshlpatel
Communicator

Hi, I have single file where I am writing all my logs and its size reached to 300 MB.

I am seeing that some logs that present in log file are not seeing in splunk.

Here my question is size of log file is impact anyway to splunk to monitor same file ?

Tags (1)
0 Karma

yannK
Splunk Employee
Splunk Employee

Size should not impact, but will be a bit slower if you have many large files or zip files to monitor.

Do you see gaps in the logs, or do you see the file monitoring stopping ?

  • Check in your splunkd.log if the file is not skipped for another reason (CRC identical to another file, or permission change)
  • also search over all time for this source, in case the problem is that the timestamp of the events is missinterpreted (timezone, eventin the future, or event far in the past)

rameshlpatel
Communicator

i am seeing some logs are not coming in splunk where its present in log file.

0 Karma

lguinn2
Legend

Did you check the splunkd.log file for error or warning messages?

0 Karma

yannK
Splunk Employee
Splunk Employee

Actually I heard of a bug at one point when the file size grows between the limit between the tailingprocessor (small files changing quickly) and the batchprocessor (large files not changing often).
Try to restart the forwarder and see if it cause the end of the file to be indexed.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...