Customer reported that after enabling rlog.sh on Ubuntu Universal Forwarder they were getting 4 x the number of events they were expecting. From a 500,000 line audit.log, they received over 2 million events in about 10 minutes. Further investigation showed that events were being duplicated by a factor of 4.
Audit.log is read by a scripted input ( rlog.sh
) which runs every 60 secs
The script keeps is own seek_pointer ( unix_audit_seekfile
) in $SPLUNK_HOME/var/run/splunk
The problem is caused by the fact that the script runs under #!/bin/sh
Part of the code tests for the existence of unix_audit_seekfile
with [ -a $SEEK_FILE ]
If the file does not exist, we create a new pointer " 0
"
On most systems /bin/sh
is a link to /bin/bash
BUT on Ubuntu it is a link to /bin/dash
[ -a $SEEK_FILE ]
will fail silently when running under /bin/dash
as " test -a
" is not a supported option.
As the "test for seekfile" now fails (even though the file exists), a new seekfile is created with " 0
"
This caused the entire audit.log to be re-read on each iteration
The work-around is to change the first line of the script from #!/bin/sh
to #!/bin/bash
to ensure bash shell execution.
A defect ticket has been raised with development to rectify this issue (TAG-9906)