The edits I made to props.conf and transforms.conf are not working to filter my data. Can anyone check to see where I went wrong?

[snehalk]

Hello All,

I have written the below props.conf and transforms.conf files, but am not able to filter my data, could anyone please let me know where am wrong?

my sample input file contains

*888*12223322*13~
Gaa7*12*216621~
ABC***1212121*470.01**2121212~
ML*14442126616682872*12~
ML*1442123133552772*12~
ML*1214241161127711722*12~
ML*121221111728182112122*12~
AMT*DD*250~
LL*11001*
XTP*434*D8*20151028~
LX*1~
LL*110016686*
XTP*434*yD8*20151028~
LL*11001*
XTP*573*D8*20161028~
LX*2~
LL*11001*
XTP*4134*D8*20161028~
LL*11001*
XTP*573*D8*20161028~
LX*3~
LL*11001*
XTP*434*D8*20161028~
LL*11001*
XTP*573*D8*20161028~
LX*4~
LL*11001*
XTP*434*D8*20161028~

what i want in output is as follows ( each event need to start with "ABC")

ABC***1212121*470.01**2121212~
AMT*DD*250~
LX*1~
XTP*573*D8*20161028~
LX*2~
XTP*573*D8*20161028~
LX*3~
XTP*573*D8*20161028~
LX*4~

Below are my configuration files

inputs.conf

[monitor://C:\trash\*.txt]
index=xyz
sourcetype=regdata

transforms.conf

[regdataparse]
REGEX = ^(ABC.+|AMT.+|LX.+|XTP\*573\*.+)
FORMAT = $1
DEST_KEY = _raw

props.conf

[regdata]
TRANSFORMS-regdataparses=regdataparse

And all files are in search app local directory (C:\Program Files\Splunk\etc\apps\search\local) and also the above regular expression is working on link text online tool

Please help me here.

[gcusello · Accepted solution]

Hi snehalk,
you have to modify the REGEX row in your transform.conf:

REGEX = ABC|AMT|LX|XTP\*573\*

inserting the strings you want divided by pipe (|)

In addition I suggest to you to put your conf files in a dedicated App and not using Search App.

Bye.
Giuseppe

[snehalk]

Hello cusello,

Am getting "$1" as log entry in splunk. please help me, shall i remove FORMAT = $1 ?

[gcusello]

Sorry I was concentrated on the regex, but there is another problem:

[regdataparse]
REGEX = ABC|AMT|LX|XTP\*573\*
DEST_KEY = queue
FORMAT = indexQueue

Bye.
Giuseppe

[snehalk]

Hello Giuseppe,

still the filter is not working, it pulling all raw data, can you please help me here?

[gcusello]

I usually use this configuration to filter my logs!
Every way I ask you some stupid questions:

• did you restarted Splunk after file modify?
• where you inserted this three conf files? they have to be on the Indexers. This is a configuration example that I used to filter Windows logs

[set_filter]
REGEX = EventCode=4624|EventCode=4625|EventCode=528|EventCode=529|EventCode=4647|EventCode=4634|EventCode=551|EventCode=538|EventCode=4727|EventCode=4737|EventCode=4728|EventCode=4729|EventCode=4730|EventCode=4754|EventCode=4755|EventCode=4756|EventCode=4757|EventCode=4758|EventCode=4731|EventCode=4735|EventCode=4732|EventCode=4733|EventCode=4734|EventCode=4720|EventCode=4724|EventCode=4738|EventCode=4725|EventCode=4722|EventCode=4726|EventCode=4719|EventCode=4781|EventCode=4764
DEST_KEY = queue
FORMAT = indexQueue

Bye.
Giuseppe

[snehalk]

Hello Giuseppe,

Yes, i restarted the splunk, whenever i did change in configuration file, and right know am doing in my local machine, once it get parse will put in our prod indexer.

Configuration file path is follows
C:\Program Files\Splunk\etc\apps\Mycustapp\local

Please help me here!!

[gcusello]

Ok the problem is that you have to parse the requested logs and discard the others, modify your conf files in this way:
props.conf

[regdata]
TRANSFORMS-regdataparses=set_nullqueue,regdataparse

transforms.conf

[regdataparse]
REGEX = ABC|AMT|LX|XTP\*573\*
DEST_KEY = queue
FORMAT = indexQueue
[set_nullqueue]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

It's important the order in TRANSFORMS command:

• if you want to take some logs and discard the others, insert before set_nullqueue and then your filter:
• if you want to discard some logs and take the others, insert before your filter and then set_nullqueue:

see http://docs.splunk.com/Documentation/Splunk/6.5.1/Forwarding/Routeandfilterdatad

Bye.
Giuseppe

[snehalk]

Hello Giuseppe,

Thanks for your help, additional to above props.conf file, i have added one more line and its working fine .

my updated props.conf

[regdata]
SHOULD_LINEMERGE = false
TRANSFORMS-regdataparses=set_nullqueue,regdataparse

Transforms.conf

[regdataparse]
 REGEX = ABC|AMT|LX|XTP\*573\*
 DEST_KEY = queue
 FORMAT = indexQueue

[set_nullqueue]
 REGEX=.
 DEST_KEY=queue
 FORMAT=nullQueue

Thank you once again!!