The Splunk Universal Forwarder 6.5.1 seems to skip the data added to the log file, once the splunk service was not running.
Problem: Forwarder is configured to forward and index the logs of some custom Java based application. It does this correctly, until the service is stopped by any reason, i.e. system update. Should the custom application write any additional data to the log, after the forwarder is stopped, those records will not be picked up after the service restart,
Relevant stanza looks like this (allowing logs of the the few apps for given source-type) in inputs.conf:
[monitor:///var/log/splunk/custom/*/<SOURCETYPE_NAME>/...]
sourcetype=<SOURCETYPE_NAME>
whitelist = (a|b|c).*
followSymlink = true
I know that Splunk forwarder does not reindex the files by default and that it uses CRC for handling the rolling log files
(http://docs.splunk.com/Documentation/Splunk/6.0.1/Data/Howlogfilerotationishandled)
It seems that only the CRC for the begining of the file is checked, once the service is running again. In the logs I even see that after starting back, it checks the file:
08-22-2018 18:31:22.655 +0200 INFO WatchedFile - Will begin reading at offset=91875 for file='/var/log/splunk/custom/<INDEX_NAME>/<SOURCETYPE_NAME>/logs/c.2018-08-22.2.log'.
The offset mentioned is exactly the last offset the service has seen before being stopped ( checked with linux dd utility), but I do not see the data starting from that offset in the search results.
Question: How can I make sure that the data added to that log file during the period of splunk outage, actually gets indexed, and provisioned during the searches?
... View more