All Apps and Add-ons

Why does Splunk_TA_nix rlog.sh cause huge amount of duplicate audit events on Ubuntu?

dshakespeare_sp
Splunk Employee
Splunk Employee

Customer reported that after enabling rlog.sh on Ubuntu Universal Forwarder they were getting 4 x the number of events they were expecting. From a 500,000 line audit.log, they received over 2 million events in about 10 minutes. Further investigation showed that events were being duplicated by a factor of 4.

dshakespeare_sp
Splunk Employee
Splunk Employee

Audit.log is read by a scripted input ( rlog.sh ) which runs every 60 secs
The script keeps is own seek_pointer ( unix_audit_seekfile ) in $SPLUNK_HOME/var/run/splunk
The problem is caused by the fact that the script runs under #!/bin/sh
Part of the code tests for the existence of unix_audit_seekfile with [ -a $SEEK_FILE ]
If the file does not exist, we create a new pointer " 0 "

On most systems /bin/sh is a link to /bin/bash BUT on Ubuntu it is a link to /bin/dash
[ -a $SEEK_FILE ] will fail silently when running under /bin/dash as " test -a " is not a supported option.
As the "test for seekfile" now fails (even though the file exists), a new seekfile is created with " 0 "
This caused the entire audit.log to be re-read on each iteration

The work-around is to change the first line of the script from #!/bin/sh to #!/bin/bash to ensure bash shell execution.
A defect ticket has been raised with development to rectify this issue (TAG-9906)

Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...