I have following set up in props.conf and transforms.conf.
props.conf
[source::/opt/apps/splunk/data/test/*]
TRANSFORMS-set= setnull,setparsing
transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = (?:Mkt.*\n)(?s)(.*?)(?:[A-Z])
DEST_KEY = queue
FORMAT = indexQueue
I am ingesting a data file containing following data through a universal forawrder, and I made the above change in /splunk/etc/system/local
:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sample ABC Report
Date:2009-12-01
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
FileName: xxx.txt
-------------------------------------------------------------------
File Processing Start Time:20091201 07:12:14
This file contains the following payments:
Mkt Bk Sender Id Cntry Curr Total Value Total Records
001 0871 1234 124 124 1140843 7
001 0700 2341 124 124 399559 2
001 9875 4567 124 124 7373838 76
001 0700 6789 124 124 761825 5
ABC.EM.0072.0005
ABC.TM.0072.0006
CDE.EM.0098.0007
File Processing End Time:20091201 07:12:14
-------------------------------------------------------------------
And my objective is to select only following data from above
001 0871 1234 124 124 1140843 7
001 0700 2341 124 124 399559 2
001 9875 4567 124 124 13842408 76
001 0700 6789 124 124 761825 5
I checked with https://regex101.com/ and my regular expression is extracting the same.
Please advise what could be possible reason that I am getting all data without filtering it. I restarted the indexer after the change.
I suspect that the source is not matching the source specified in your props.conf
- remember that the source is case sensitive.
[source::/opt/apps/splunk/data/test/*]
TRANSFORMS-set=setnull,setparsing
However, if it was working, you would have another problem. The setnull
transform will send all the events to the nullQueue
, thereby deleting them. Why? Because the REGEX (.
) matches every event - and you have specified that this transform is applied first. Second, the REGEX is applied to each event individually, while http://regex101.com/ is looking at the text as a whole.
I think this will work better, assuming that you get the source
correct:
props.conf
[source::/opt/apps/splunk/data/test/*]
SHOULD_LINEMERGE=false
TRANSFORMS-set= setparsing,setnull
transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = ^\s*\d{3}\s+\d{4}
DEST_KEY = queue
FORMAT = indexQueue
Location is Correct. When I was using my configuration no filtering was happening but now with you set up no data is coming I mean data is not ingesting at all..... as data is not coming so it confirms that source is matching rest condition blocking the data.
Please advise.
The props/transforms are placed on Indexers, correct?
yes I kept in indexer