Splunk Search

Data is not getting filtered with my set regex in transforms.conf?

sandipan11
Path Finder

I have following set up in props.conf and transforms.conf.

props.conf

[source::/opt/apps/splunk/data/test/*]
TRANSFORMS-set= setnull,setparsing

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (?:Mkt.*\n)(?s)(.*?)(?:[A-Z])
DEST_KEY = queue
FORMAT = indexQueue

I am ingesting a data file containing following data through a universal forawrder, and I made the above change in /splunk/etc/system/local:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                Sample ABC Report   
                Date:2009-12-01     
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
FileName: xxx.txt
-------------------------------------------------------------------
File Processing Start Time:20091201 07:12:14

This file contains the following payments:                                      
Mkt  Bk    Sender Id    Cntry  Curr         Total Value          Total Records
001  0871  1234          124    124             1140843                  7
001  0700  2341          124    124              399559                  2
001  9875  4567          124    124            7373838                  76
001  0700  6789          124    124              761825                  5
ABC.EM.0072.0005
ABC.TM.0072.0006
CDE.EM.0098.0007

File Processing End Time:20091201 07:12:14
-------------------------------------------------------------------

And my objective is to select only following data from above

001  0871  1234          124    124             1140843                  7
001  0700  2341          124    124              399559                  2
001  9875  4567          124    124            13842408                  76
001  0700  6789          124    124              761825                  5

I checked with https://regex101.com/ and my regular expression is extracting the same.

Please advise what could be possible reason that I am getting all data without filtering it. I restarted the indexer after the change.

0 Karma

lguinn2
Legend

I suspect that the source is not matching the source specified in your props.conf - remember that the source is case sensitive.

[source::/opt/apps/splunk/data/test/*]
TRANSFORMS-set=setnull,setparsing

However, if it was working, you would have another problem. The setnull transform will send all the events to the nullQueue, thereby deleting them. Why? Because the REGEX (.) matches every event - and you have specified that this transform is applied first. Second, the REGEX is applied to each event individually, while http://regex101.com/ is looking at the text as a whole.

I think this will work better, assuming that you get the source correct:

props.conf

[source::/opt/apps/splunk/data/test/*]
SHOULD_LINEMERGE=false
TRANSFORMS-set= setparsing,setnull

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^\s*\d{3}\s+\d{4}
DEST_KEY = queue
FORMAT = indexQueue
0 Karma

sandipan11
Path Finder

Location is Correct. When I was using my configuration no filtering was happening but now with you set up no data is coming I mean data is not ingesting at all..... as data is not coming so it confirms that source is matching rest condition blocking the data.

Please advise.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

The props/transforms are placed on Indexers, correct?

0 Karma

sandipan11
Path Finder

yes I kept in indexer

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...