Hello
The way the script is constructed makes it consume a lot of CPU when audit logs are big (100M). Because it reads the whole file, skipping line by line.
I have this proposal:
TAIL_SIZE=$((FILE_LINES-SEEK))
if [ $TAIL_SIZE -gt 0 ]; then
exec 3<&0
exec 0<`tail -$TAIL_SIZE $AUDIT_FILE`
while read -r line
do
echo $line | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v '^----'
done
exec 0<&3
fi
This way resource usage is quite lower.
will this script be updated anytime soon?
regards,
gabriel
I am linux newbie. What information this script suppose add? We're getting an error severity with "<no matches>" what does that mean?
Here's an additional optimization that also solve the UI issue where users get the error:
msg="A script exited abnormally" input="./bin/rlog.sh" stanza="default" status="exited with code 1"
This is based on Splunk more recent releases of this script, but changes 2 things.
It swaps out the super user check with a file readability check. This makes it possible to run as the "splunk" (privileged) user as long as you've setup file system permissions appropriately.
if [ "x$KERNEL" = "xLinux" ] ; then
#assertInvokerIsSuperuser
test -r "$AUDIT_FILE" || exit 1
assertHaveCommand service
assertHaveCommandGivenPath /sbin/ausearch
if [ -n "service auditd status
" -a "$?" -eq 0 ] ; then
if [ -a $SEEK_FILE ] ; then
SEEK=head -1 $SEEK_FILE
else
SEEK=1
echo "0" > $SEEK_FILE
fi
FILE_LINES=wc -l $AUDIT_FILE | cut -d " " -f 1
if [ $FILE_LINES -eq $SEEK ] ; then
# No new events in audit.log
exit 0
fi
if [ $FILE_LINES -lt $SEEK ] ; then
# audit file has wrapped
SEEK=0
fi
awk -v START=$SEEK -v OUTPUT=$SEEK_FILE 'NR>START { print } END { print NR > OUTPUT }' $AUDIT_FILE | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v "^----"
fi
Just in case somebody stumbles across this some times and tries to copy-paste the script.
It seems the line
if [ -a $SEEK_FILE ] ; then
needs to be replaced with
if [ -e $SEEK_FILE ] ; then
As the "-a" does not seem to work with newer bash versions.
Thank you for your contribution, but I would no longer suggest to use my version of the script, because Splunk already updated their script in their current app.
Good answers / workarounds. This is now officially fixed by Splunk in the *Nix App 5.0 -- http://apps.splunk.com/app/833/
We ran into the same problem and extended the previous solutions.
Now we do not even use a loop at all.
We have to note thought, that the script now depends on tail being installed.
. `dirname $0`/common.sh
SEEK_FILE=$SPLUNK_HOME/var/run/splunk/unix_audit_seekfile
AUDIT_FILE=/var/log/audit/audit.log
if [ "x$KERNEL" = "xLinux" ] ; then
assertInvokerIsSuperuser
assertHaveCommand service
assertHaveCommandGivenPath /sbin/ausearch
if [ -n "`service auditd status`" -a "$?" -eq 0 ] ; then
if [ -a $SEEK_FILE ] ; then
SEEK=`head -1 $SEEK_FILE`
else
SEEK=0
echo "0" > $SEEK_FILE
fi
FILE_LINES=`wc -l $AUDIT_FILE | cut -d " " -f 1`
if [ $FILE_LINES -lt $SEEK ] ; then
# audit file has wrapped
SEEK=0
fi
wc -l $AUDIT_FILE | cut -d " " -f 1 > $SEEK_FILE
exec 3<&0
exec 0<"$AUDIT_FILE"
tail -n +$(($SEEK+1)) | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v '^----'
exec 0<&3
fi
elif [ "x$KERNEL" = "xSunOS" ] ; then
:
elif [ "x$KERNEL" = "xDarwin" ] ; then
:
elif [ "x$KERNEL" = "xFreeBSD" ] ; then
:
fi
I would no longer suggest to use my version of the script, because Splunk already updated their script in their current app.
In my case the high CPU seemed to be coming from the the COUNT and SEEK . So I commented out several lines and CPU returned to normal. Here is a copy of my rlog.sh
#!/bin/sh
# Copyright 2011 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
. `dirname $0`/common.sh
SEEK_FILE=$SPLUNK_HOME/var/run/splunk/unix_audit_seekfile
AUDIT_FILE=/var/log/audit/audit.log
if [ "x$KERNEL" = "xLinux" ] ; then
assertInvokerIsSuperuser
assertHaveCommand service
assertHaveCommandGivenPath /sbin/ausearch
if [ -n "`service auditd status`" -a "$?" -eq 0 ] ; then
if [ -a $SEEK_FILE ] ; then
SEEK=`head -1 $SEEK_FILE`
else
SEEK=1
echo "1" > $SEEK_FILE
fi
FILE_LINES=`wc -l $AUDIT_FILE | cut -d " " -f 1`
if [ $FILE_LINES -lt $SEEK ] ; then
# audit file has wrapped
SEEK=1
fi
exec 3<&0
exec 0<"$AUDIT_FILE"
##-# COUNT=0
##-# while read -r line
tail -n +$SEEK | while read -r line
do
##-# if [ $COUNT -lt $SEEK ] ; then
##-# COUNT=`expr $COUNT + 1`
##-# else
echo $line | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v '^----'
##-# COUNT=`expr $COUNT + 1`
##-# fi
done
exec 0<&3
##-# echo $COUNT > $SEEK_FILE
wc -l $AUDIT_FILE | cut -d " " -f 1 > $SEEK_FILE
fi
elif [ "x$KERNEL" = "xSunOS" ] ; then
:
elif [ "x$KERNEL" = "xDarwin" ] ; then
:
elif [ "x$KERNEL" = "xFreeBSD" ] ; then
:
fi
So at this point it looks like there are a couple different solutions to this -- Your (JSapienza) solution regarding count and seek, and the original issue/solution submitted by gdiazlo. I wonder if we can get some input from splunk as to their thoughts on this; the *nix app is their app after all. I'd be interested to hear their take...
Perhaps I'll ask them as I believe I have access to support 🙂
Thanks for the information and quick response!
correction:
TAIL_SIZE=$((FILE_LINES-SEEK))
if [ $TAIL_SIZE -gt 0 ]; then
tail -$TAIL_SIZE $AUDIT_FILE | while read -r line
do
echo $line | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v '^----'
done
echo $FILE_LINES > $SEEK_FILE
fi
as the other sintaxs wasn't correct.
I have also noticed high CPU usage on my universal forwarders that is coming from rlog.sh. Has there been an official fix/patch for this?
Right now our splunk is running in a dev/testing environment/configuration, so I was probably going to just temporarily disable the rlog.sh script until we can work this out.
So any ideas/suggestions/fixes/confirmations of fixes would be great!
thx
I am having the same issue, Was this correction successful ?