Hi Splunkers, I got a requirement from the cyber security user to remove the events with header displaying empty and retain those events with header details having some values into it.
Header Details: USERNAME LINE HOSTNAME TIME
We are using Splunk Add-on for Unix and Linux version 5.2.4 to capture the UNIX details in splunk and the who.sh scripted input are used to capture the who logged on to the system details and both the Linux, Solaris OS details are successfully ingested into splunk via scripted inputs but there are many events with zero values in it and only Header details are shown as a event which we wanted to remove it.
Attached snap shot for your references.
. dirname $0
/common.sh
CMD='who -H'
HEADER='USERNAME LINE HOSTNAME TIME'
HEADERIZE='{NR == 1 && $0 = header}'
FORMAT='{length(hostname) || hostname=$NF; gsub("[)(]", "",hostname); time=$3; for (i=4; i<=lastTimeColumn; i++) time = time " " $i}'
PRINTF='{if (NR == 1) {print $0} else {printf "%-14s %-10s %-40.40s %-s\n", $1,$2,hostname,time}}'
if [ "x$KERNEL" = "xLinux" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 5) {hostname = "
elif [ "x$KERNEL" = "xSunOS" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 6) {hostname = "
elif [ "x$KERNEL" = "xAIX" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 6) {hostname = "
elif [ "x$KERNEL" = "xDarwin" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 6) {hostname = "
elif [ "x$KERNEL" = "xFreeBSD" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 6) {hostname = "
fi
assertHaveCommand $CMD
$CMD | tee $TEE_DEST | $AWK "$HEADERIZE $FILL_BLANKS $FORMAT $PRINTF" header="$HEADER"
echo "Cmd = [$CMD]; | $AWK '$HEADERIZE $FILL_BLANKS $FORMAT $PRINTF' header=\"$HEADER\"" >> $TEE_DEST
Note: we have used Line_Breaker stanza in props.conf to break the multiple events into single events as per the requirement.
[who]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
TRUNCATE=1000000
DATETIME_CONFIG = CURRENT
Kindly guide us on how to do this customization.
Make sure the addon is also deployed on the indexers, search heads, and any heavy forwards in the env. I once experienced this and it was showing up merely because I forgot to follow the documentation regarding where to deploy the TA and so the sourcetypes were not kicking in correctly.
Can anyone throw me some lights on this issues ?
You will have to write props.conf and transforms.conf to remove unwatered rows from your logs.
If you have HF,
Add the below props.conf in your search app local folder.
[yoursourcetype]
TRANSFORMS-set = setnull
And create transforms.conf in the same folder
And add below lines to it
[setnull]
REGEX = USERNAME
DEST_KEY = queue
FORMAT = nullQueue
If you dont have heavy forwarder add this in indexer and restart splunkd.
Hi Thambisetty, thanks for your support, I think I have tried to this but it did not work out, yes we have HF for parsing the data before indexing the data.
Props:
[yoursourcetype]
TRANSFORMS-set= setnull,setparsing
Transforms.conf :
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
basically keeping anything that has header and at least one data row
[setparsing]
REGEX = ^(USERNAME).+[\r\n]+\w+
DEST_KEY = queue
FORMAT = indexQueue
Anyway let me try it once again.