Getting Data In

Will Splunk re-start processing of a file if it is renamed?

Brian
Engager

I am monitoring a directory with contains files that are rotated.

Example:

A file, today.logs is currently being processed by splunk

tomorrow, the today.logs will be renamed to yesterday.logs

My question is: will splunk process these events twice?

Tags (1)

Lowell
Super Champion

Splunk should detect renames files like this. In fact, if Splunk wasn't completely done indexing everything in "today.log" when it was renamed to "yesterday.log", then it will read the remaining portion of "yesterday.log" until it's caught up.

The feature works by recording hash values (finger prints) of the beginning and ending of the files being monitored, this way it can detect when a file was renamed. This also lets splunk recolonize when the file "today.log" has been truncated and new content is being written to the same file name, in which case splunk will read the new events starting at the beginning of the log index them.

Please note that you can alter (often just "break") this functionality by setting a crcSalt value in your monitor input stanza; which I recommend staying away from if at all possible. So, unless you've explicitly added such a setting, splunk should "just work" with your log files -- each event should be indexed once and only once.

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...