Splunk Search

Forcing LWF to send events from a file

beaumaris
Communicator

I have a system configured with a separate search head, indexer and LWF. In order to validate our processing and refine some of the filtering of comment lines using REGEX, I want to be able to run a test suite multiple times using a test file deposited to a directory on the LWF that it monitors. The plan is to use the 'clean' command on the indexer in between runs, and then remove the test file and copy it again to the LWF to start the input/parse cycle again. However even if I rename the file splunk seems to figure out it's the same contents, and the events are not sent to the Indexer. Log messages from splunkd.log are similar to the following

01-29-2011 08:02:32.730 ERROR TailingProcessor - Ignoring path due to: File will not be read, is too small to match seekptr checksum (file=/home/cdndata/we_accesslog_extsqu_69.241.50.154_20110127_080000_02272_A.gz). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info. 01-29-2011 08:02:34.729 INFO TailingProcessor - Archive file='/home/cdndata/we_accesslog_extsqu_69.241.50.182_20110126_200000_03855_A.gz' updated less than 10000ms ago, will not read it until it stops changing.

Is there a way to force the LWF to read the file and send the events, or are there any tricks we can use to make Splunk think it's a new file? We really need to keep the events constant as we proceed with refining the index, summarization and reporting parts of the system. Thanks!

Tags (1)

chris
Motivator

Hi

Splunk keeps track of what it has already indexed by building crc sums of the beginning and the end of files if another file matches those sums even if its name is different it will not be reindexed

http://www.splunk.com/base/Documentation/latest/Admin/Howlogfilerotationishandled

There is a parameter crcSalt that you can add to your monitored file in inputs.conf to overcome this behaviour. If the parameter is set to the special value <SOURCE> the full path to the file will be added to the crc sum and every time you rename the file it should get reindexed.

[monitor:///path/to/file/xyz]
index = myIndex
crcSalt = <SOURCE> 

Good luck

Chris

0 Karma

beaumaris
Communicator

Great explanation and reference, thanks Chris!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...