Splunk Enterprise

splunk duplicate events after forwarder running in a container gets restarted

rajholla_optum
Engager

We have splunk forwarder running in a docker container and all our workloads which is also running in different containers and writes logs to NFS file mounts on dedicated location.
The problem here is , when container running forwarder restarts it simply sees all file as new and reads them again causing duplicate events.
I assume the problem here is, when forwarder starts in container it becomes new installation of a forwarder.
Can this be solved by persisting forwarder file system(/opt/splunk/splunkforworder/*) ? Or is there any alternative ?

Tags (1)
0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

if persisting the forwarder is possible (im not a docker guy) then it would solve the issue as Splunk would then remember where it was in each of the files. check out the fishbucket!

https://www.splunk.com/blog/2008/08/14/what-is-this-fishbucket-thing/

- MattyMo

View solution in original post

rfaircloth_splu
Splunk Employee
Splunk Employee

I would also be very concerned the NFS portion of this solution would allow for loss of logs if a close happens without the proper time for flushing to occur. Normally streaming to remote mounts is considered a bad practice. Consider the use of the splunk docker log driver for OS logs and other means of streaming delivery such rsyslog or HEC to avoid the file IO.

rajholla_optum
Engager

After persisting "/opt/splunk/splunkforwarder" forwarder was able to read the files at specific offset.
Information about fishbucket was very really helpful.

Thank you!

0 Karma

mattymo
Splunk Employee
Splunk Employee

if persisting the forwarder is possible (im not a docker guy) then it would solve the issue as Splunk would then remember where it was in each of the files. check out the fishbucket!

https://www.splunk.com/blog/2008/08/14/what-is-this-fishbucket-thing/

- MattyMo

MuS
SplunkTrust
SplunkTrust

Is the Splunk forwarder reading from NFS or writing its logs to NFS?
Check any existing inputs.conf for an option called crcSalt if this is set it can produce duplicate events.

0 Karma

rajholla_optum
Engager

Thank you for your time !

Forwarder reading log files from NFS.
Forwarder writes everything into container write layer which wont be persisted after restart.
We are not using crcSlat in our inputs.conf

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...