We have splunk forwarder running in a docker container and all our workloads which is also running in different containers and writes logs to NFS file mounts on dedicated location.
The problem here is , when container running forwarder restarts it simply sees all file as new and reads them again causing duplicate events.
I assume the problem here is, when forwarder starts in container it becomes new installation of a forwarder.
Can this be solved by persisting forwarder file system(/opt/splunk/splunkforworder/*) ? Or is there any alternative ?
if persisting the forwarder is possible (im not a docker guy) then it would solve the issue as Splunk would then remember where it was in each of the files. check out the fishbucket!
https://www.splunk.com/blog/2008/08/14/what-is-this-fishbucket-thing/
I would also be very concerned the NFS portion of this solution would allow for loss of logs if a close happens without the proper time for flushing to occur. Normally streaming to remote mounts is considered a bad practice. Consider the use of the splunk docker log driver for OS logs and other means of streaming delivery such rsyslog or HEC to avoid the file IO.
After persisting "/opt/splunk/splunkforwarder" forwarder was able to read the files at specific offset.
Information about fishbucket was very really helpful.
Thank you!
if persisting the forwarder is possible (im not a docker guy) then it would solve the issue as Splunk would then remember where it was in each of the files. check out the fishbucket!
https://www.splunk.com/blog/2008/08/14/what-is-this-fishbucket-thing/
Is the Splunk forwarder reading from NFS or writing its logs to NFS?
Check any existing inputs.conf
for an option called crcSalt
if this is set it can produce duplicate events.
Thank you for your time !
Forwarder reading log files from NFS.
Forwarder writes everything into container write layer which wont be persisted after restart.
We are not using crcSlat in our inputs.conf