Getting Data In

Will monitored files that are renamed be reindexed?

joshuapetitt
Path Finder

I have an application that uses rolling logging. The rolling logging works as such:

  • logs are initially written to "log_file.txt"
  • after a certain time period, the "log_file.txt" is renamed to "log_file.txt.1", and an empty "log_file.txt" is created to start writing to.
  • after a certain time period, the "log_file.txt.1" is renamed to "log_file.txt.2", "log_file.txt" is renamed to "log_file.txt.1" and an empty "log_file.txt" is created to start writing to.
  • this process continues until X number of files are written, after that the oldest file file is deleted.

So if I monitor all of the files that match "log_file.*", will I have a bunch of duplicated data each time the file rolls over?

Should I just monitor "log_file.txt" in this instance?

0 Karma
1 Solution

micahkemp
Champion

The answer is somewhat "it depends", but the default settings are likely sane enough to prevent re-indexing old data. Splunk uses the first initCrcLength bytes of a file to determine if it is a new file or not. That setting can be changed as necessary, and other settings can also be changed to force Splunk to use the filename to assist it in determining uniqueness, but it sounds like you want the normal behavior that a monitor input would give you.

For more information on the different settings you can use for your inputs check out the inputs.conf spec.

View solution in original post

micahkemp
Champion

The answer is somewhat "it depends", but the default settings are likely sane enough to prevent re-indexing old data. Splunk uses the first initCrcLength bytes of a file to determine if it is a new file or not. That setting can be changed as necessary, and other settings can also be changed to force Splunk to use the filename to assist it in determining uniqueness, but it sounds like you want the normal behavior that a monitor input would give you.

For more information on the different settings you can use for your inputs check out the inputs.conf spec.

MuS
SplunkTrust
SplunkTrust

Did you mean inputs.conf http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf ?
props.conf only has initCrcLength but anything else is configurable in inputs.conf

cheers, MuS

0 Karma

micahkemp
Champion

Yes, thanks! I had linked to the correct page but put props for the text instead.

0 Karma

woodcock
Esteemed Legend

It also very much depends on whether you are using crcSalt = <SOURCE> or not.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...