Getting Data In

How to filter the data in my log before indexing?

Tejkumar451
Explorer

HI I have a scenario where I need to filter the data present in a log. I need to index only the last line from that log into splunk & ignore the rest of the lines. I am using a universal forwarder here.

Example file: abc 
---------------------------------------------------------------
abcdefgh
bda
cdb
---
---
---
Splunk000012345
--------------------------------------------------------------

So, I have to ingest only the last line "Splunk000012345" into Splunk ignore the above lines. And just an fyi, I am even fine if I can ingest only the number 000012345 too.

I did the below changes on the forwarder side, but it didn't work.

Edit props.conf and add the following:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

Edit transforms.conf and add the following:

[setnull]
REGEX = .
DEST_KEY = queue

[setparsing]
REGEX = Splunk(?[^\w])
DEST_KEY = queue
FORMAT = indexQueue
0 Karma

Tejkumar451
Explorer

Forgot to add one thing, I did all this on the forwarder side & all the lines are getting ingested which shouldn't be the case.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

You can test regex's out over at regex.com and see what they do or don't pull. Here's your problem -

Splunk(?[^\w])

The above regex translates as the letters "Splunk" followed by a single character that is NOT a "word" character. That is, the first character after Splunk must be neither a digit nor an upper or lower case letter. Might exclude underscores or hyphens or something as well, I'd have to look it up. In any case, your zero will fail to match "not being a word character" so the line will not be ingested.

Try this -

Splunk(?\d+)
0 Karma

Tejkumar451
Explorer

Thanks for the response, I have tested this regex Splunk(\d+) in regex101.com and its working fine over there. So, now my concern is should these props & Transforms.conf be done in the Universal Forwarder end? or towards the indexer side? As I am not exactly sure if parsing can be done on the Universal forwarder side.

Thanks

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...