Getting Data In

If I have a monitored log file with lines that are overwritten rather than appended, will this cause any problems to indexing?

venkat_d
New Member

Suppose I am monitoring a file that has 10 lines in it. These lines are already sent to splunk.
Now, if I overwrite this file to have just 5 lines (in other words, my logwriter cleans the log file every time and write some new lines in it - we are reusing the log file that is being monitored)

Will this cause any problems to splunk indexing?

Tags (3)
0 Karma

stephane_cyrill
Builder
0 Karma

stephane_cyrill
Builder

here is two scenario for you:

  1. The file is deleted or truncated and new data is rewritten from the start; or
  2. The file is written over the beginning with the same old contents up to the point where it was before, then a couple of new lines are added. In the first case, Splunk will have no problems detecting the new data. In the second case, unless the old data is written faster than Splunk can detect that it has been changed/deleted, it will probably wind up double-indexing the old data. If the old file is rewritten fast enough (or moved/renamed over the old one) then there won't be any problems
0 Karma

NOUMSSI
Builder

Hi venkat;

If when you added your file to splunk, you have choose to upload the file, any time that you'll have updates for that file, you'll need to re-indexing the new file for splunk to take care of the update.

But if you choose to monitor the file (continuously index data from file and directory), you'll not need to re-index your file; you'll just need to overwrite the old file by the new file at the same location.
NB: The new file most have the same name like the old file.

In your case, choose the option that correspond to your needs.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...