How to configure in props/transforms.conf to remove the event data which does not contain any information in it?

[Hemnaath]

Hi Splunk experts,

Just want to know how can I remove events which does not contain any information in it?
Example

sample events which have information on them.

I know that, we have to configure Route and Filter data in Props/transforms to achieve this but not sure about the Regex.

Props.conf details:
[who]
TRANSFORMS-null= setnull

Transforms.conf details:
[setnull]
 REGEX = 
 DEST_KEY = queue
 FORMAT = nullQueue

Kindly guide me on this

[somesoni2]

Give this a try (changed strategy from 'Drop specific events' to 'Keep specific event and drop remaining').
Reference: http://docs.splunk.com/Documentation/Splunk/7.1.1/Forwarding/Routeandfilterdatad#Keep_specific_event...

Edit props.conf and add the following:

[who]
TRANSFORMS-set= setnull,setparsing

Edit transforms.conf and add the following:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

#basically keeping anything that has header and at least one data row
[setparsing]
REGEX = ^(USERNAME).+[\r\n]+\w+
DEST_KEY = queue
FORMAT = indexQueue

[Hemnaath]

Hi SomeSoni2,

Yes I had given a try with the above set of configuration in my test environment by uploading the data in raw text format via Add data -->upload --> select sourcetype but it did not fetch the required output.

Instead getting the below message.
No results found. Please change source type, adjust source type settings, or check your source file

Props.conf details:
[who]
  SHOULD_LINEMERGE=false
  LINE_BREAKER=([\r\n]+)
  TRUNCATE=1000000
  DATETIME_CONFIG = CURRENT
TRANSFORMS-set= setnull,setparsing

Transforms.conf details:
[setnull]
 REGEX = .
 DEST_KEY = queue
 FORMAT = nullQueue

 #basically keeping anything that has header and at least one data row
 [setparsing]
 REGEX = ^(USERNAME).+[\r\n]+\w+
 DEST_KEY = queue
 FORMAT = indexQueue

Kindly guide me on this.

[somesoni2]

YOu said you're uploading it from an instance. Did you setup these (props and transforms) on the same server/instance? If this a test environment, try setting up a monitoring (using inputs.conf) and test the ingestion, instead of using Add Data wizard.

[Hemnaath]

thanks Somesoni2, let me try that.

[Hemnaath]

Hey it did not work, could you please guide me on this

USERNAME LINE HOSTNAME TIME

host = tt3crp00 source = who sourcetype = who

6/21/18
1:11:25.000 PM

USERNAME LINE HOSTNAME TIME

solarwinds pts/21 vmswpep02.XXXX.com Jun 21 13:11

host = tt3dev00 source = who sourcetype = who

6/21/18
1:11:25.000 PM

USERNAME LINE HOSTNAME TIME
solarwinds pts/0 vmswpep05.xxxx.com 2018-06-21 13:11
solarwinds pts/1 vmswpep05.xxxx.com 2018-06-21 13:11

the above props/transforms.conf are placed in the Heavy forwarder instances.

Kindly guide me to fix this issue.

[Hemnaath]

Hi Somesoni2, by applying the above stanza in props.conf and transforms.conf, the entire event information are removed from splunk and it throws the below message.

No results found. Please change source type, adjust source type settings, or check your source file

Could you please guide me on this to events which does not contain any information in it.

[cpetterborg]

If the Question contains the example data, then perhaps something like:

REGEX = ^USERNAME\s+LINE\s+HOSTNAME\s+TIME$

[Hemnaath]

Hi cpetterborg, thanks for your effort on this, I had added the sample events containing the data, so can I use the above regex to remove events which does not contain any information in it.

[cpetterborg]

If you are making the events be multi-line events (one for each time the command is run), then you can use the SEDCMD in props.conf to remove only the header lines.

If you are making each line be a separate event (as it appeared in your question), you can use something like my suggestion.

[Hemnaath]

Hey we are splitting multiple events in to single individual events by using the line_breaker stanza in props.conf. And also we wanted to remove the events which does not contain any information in it.

1) To split multiple events in to single individual events.

[who]
 SHOULD_LINEMERGE=false
 LINE_BREAKER=([\r\n]+)
 TRUNCATE=1000000
 DATETIME_CONFIG = CURRENT

2) For removing the events which does not contain any information.

Props.conf
[who]
 TRANSFORMS-null= setnull

 Transforms.conf details:
 [setnull]
  REGEX = ^USERNAME\s+LINE\s+HOSTNAME\s+TIME$
  DEST_KEY = queue
  FORMAT = nullQueue

Kindly guide me on this.

[somesoni2]

The REGEX will depend upon how your event looks like when it has information. Could you provide some sample events which have information on them?

[Hemnaath]

Hi Somesoni2, thanks for your effort on this, we are monitoring wtmpx file from the unix machines using the Splunk Add-on for Unix. Below are the sample events contains the information.

Index=unix sourcetype=who host=*

6/20/18
3:06:05.000 PM

USERNAME LINE HOSTNAME TIME
DBB019 pts/1 w442xty1.XXXX.com Jun 19 18:40
AMM007 pts/5 yb33gnn1.XXXX.com Jun 19 08:53
host = ttbmt02 source = who sourcetype = who
6/20/18
3:03:35.000 PM

USERNAME LINE HOSTNAME TIME
DBB019 pts/1 w442xty1.XXXX.com Jun 19 18:40
AMM007 pts/5 yb33gnn1.XXXX.com Jun 19 08:53