All Apps and Add-ons

How to remove the header details with null value in it and at same time how to retain the header details with values in splunk console?

Hemnaath
Motivator

Hi Splunkers, I got a requirement from the cyber security user to remove the events with header displaying empty and retain those events with header details having some values into it.

Header Details: USERNAME LINE HOSTNAME TIME

We are using Splunk Add-on for Unix and Linux version 5.2.4 to capture the UNIX details in splunk and the who.sh scripted input are used to capture the who logged on to the system details and both the Linux, Solaris OS details are successfully ingested into splunk via scripted inputs but there are many events with zero values in it and only Header details are shown as a event which we wanted to remove it.

Attached snap shot for your references.alt text

. dirname $0/common.sh

CMD='who -H'
HEADER='USERNAME LINE HOSTNAME TIME'
HEADERIZE='{NR == 1 && $0 = header}'
FORMAT='{length(hostname) || hostname=$NF; gsub("[)(]", "",hostname); time=$3; for (i=4; i<=lastTimeColumn; i++) time = time " " $i}'
PRINTF='{if (NR == 1) {print $0} else {printf "%-14s %-10s %-40.40s %-s\n", $1,$2,hostname,time}}'

if [ "x$KERNEL" = "xLinux" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 5) {hostname = ""; lastTimeColumn = NF}}'

elif [ "x$KERNEL" = "xSunOS" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 6) {hostname = ""; lastTimeColumn = NF}}'

elif [ "x$KERNEL" = "xAIX" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 6) {hostname = ""; lastTimeColumn = NF}}'

elif [ "x$KERNEL" = "xDarwin" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 6) {hostname = ""; lastTimeColumn = NF}}'

elif [ "x$KERNEL" = "xFreeBSD" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 6) {hostname = ""; lastTimeColumn = NF}}'
fi

assertHaveCommand $CMD
$CMD | tee $TEE_DEST | $AWK "$HEADERIZE $FILL_BLANKS $FORMAT $PRINTF" header="$HEADER"
echo "Cmd = [$CMD]; | $AWK '$HEADERIZE $FILL_BLANKS $FORMAT $PRINTF' header=\"$HEADER\"" >> $TEE_DEST

Note: we have used Line_Breaker stanza in props.conf to break the multiple events into single events as per the requirement.

[who]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
TRUNCATE=1000000
DATETIME_CONFIG = CURRENT

Kindly guide us on how to do this customization.

sloshburch
Splunk Employee
Splunk Employee

Make sure the addon is also deployed on the indexers, search heads, and any heavy forwards in the env. I once experienced this and it was showing up merely because I forgot to follow the documentation regarding where to deploy the TA and so the sourcetypes were not kicking in correctly.

0 Karma

Hemnaath
Motivator

Can anyone throw me some lights on this issues ?

0 Karma

thambisetty
SplunkTrust
SplunkTrust

You will have to write props.conf and transforms.conf to remove unwatered rows from your logs.

If you have HF,

Add the below props.conf in your search app local folder.

[yoursourcetype]
TRANSFORMS-set = setnull

And create transforms.conf in the same folder
And add below lines to it

[setnull]
REGEX = USERNAME
DEST_KEY = queue
FORMAT = nullQueue

If you dont have heavy forwarder add this in indexer and restart splunkd.

————————————
If this helps, give a like below.
0 Karma

Hemnaath
Motivator

Hi Thambisetty, thanks for your support, I think I have tried to this but it did not work out, yes we have HF for parsing the data before indexing the data.

Props:
[yoursourcetype]
TRANSFORMS-set= setnull,setparsing

Transforms.conf :

Who

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

basically keeping anything that has header and at least one data row

[setparsing]
REGEX = ^(USERNAME).+[\r\n]+\w+
DEST_KEY = queue
FORMAT = indexQueue

Anyway let me try it once again.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...