Hi All,
I'm trying to do index field extractions from source files, here is the my settings
file names are like:
/tmp/test-raj/abc/bcd.log
/tmp/test-raj/xyz/cccc.log
i want to extract the 3rd directory as a fields called raj
raj=abc
raj=xyz
+
+ etc
Transforms.conf
i placed transforms.conf file in UF and HF and indexer
[netscreen-error]
SOURCE_KEY = MetaData:Source
REGEX = \/tmp\/test-raj\/(?\w+[^\/]+)\/\S+ in source
FORMAT = raj::"$1"
WRITE_META = true
Props.conf
i placed props.conf file in UF and HF and indexer
[test-123]
TRANSFORMS-netscreen = netscreen-error
fileds.conf
[raj]
INDEXED = true
let me know your thoughts?
Your regex seems incorrect
Please try in transforms.conf
[netscreen-error]
SOURCE_KEY = MetaData:Source
REGEX = \/tmp\/test-raj\/([^\/]+)\/.+
FORMAT = raj::$1
WRITE_META = true
Regex101 => https://regex101.com/r/OK0pNj/1
Hi @rajasekhar14,
In addition to fixing your regex and format as @koshyk mentioned it :
REGEX = \/tmp\/test-raj\/([^\/]+)\/.+
FORMAT = raj::$1
Make sure you add the fields.conf
file to the indexer as even if your indexed field is written in the metadata the indexer will not use it unless defined in the fields.conf
file.
Cheers,
David
@DavidHourani i placed the fileds.conf file and other files in the indexers only, but look likes it nor working. any thoughts??
yeah it depends on where your data is coming from. If its going through a Heavy Forwarder then you need props.conf and transforms.conf on the HF and fields.conf on the indexer. If there is no HF then putting all on the indexer should do the trick.
Thanks @DavidHourani
Most welcome @rajasekhar14 ! Please accept or upvote my answer and comments ❤️
Your regex seems incorrect
Please try in transforms.conf
[netscreen-error]
SOURCE_KEY = MetaData:Source
REGEX = \/tmp\/test-raj\/([^\/]+)\/.+
FORMAT = raj::$1
WRITE_META = true
Regex101 => https://regex101.com/r/OK0pNj/1
@koshyk thanks for the answer, i changed my Regex but its not working.
now all 3 files are in the only indexers
[splunk@**** local]$ cat transforms.conf
[netscreen-error]
SOURCE_KEY = MetaData:Source
REGEX = \/tmp\/test-raj\/([^\/]+)\/.+
FORMAT = raj::$1
WRITE_META = true
[splunk@**** local]$ cat props.conf
[test-123]
TRANSFORMS-netscreen = netscreen-error
[splunk@*** local]$ cat ../../spl_fields/local/fields.conf
[raj]
INDEXED = true
do i need to change these .conf to HF or UF??
if you have HF, you need to send to HF & Indexers
Need to restart HF & indexers too if possible
Hi @koshyk ,
i have a small question on this, the above settings will use for source file name right? if i want to extract a index filed extraction in side from source file,?
i changed like this but its not working. can you please take a look.
props.conf
[ms:iis:auto]
TRANSFORMS-raj_namee = test-raj
Transforms.conf
[test-raj]
REGEX = ^(?:[^ \n]* ){2}([^ ]+)
FORMAT = appname::$1
WRITE_META = true
filed.conf
INDEXED=true
and the log format is
2019-07-17 18:21:33 xx-xx.xxx test 10.185.162.2 GET /monitor/monitor.html ----
and i'm using the above regex bold text and it need extract as a appname.