I am able to extract a field using rex and sed in a search, but why is the field not extracted correctly configuring the regex in props.conf?

[splunker1981]

Hello Splunkers

I am currently using the following regex+sed to make one of my extracted fields usable. Trying to avoid having to do this by adding SEDCMD entry to my props.conf for the specific sourcetype so it gets extracted correctly the first time around. For some reason the field I am hoping to update does not change when I add the rex command used below within the config. The rex+sed commands when used in a search give me the results I want, and I end up creating a new field to be able to work with the data. Wondering if someone can help me out as I am not sure what I am doing wrong? Do I need to update more than one config file when trying to fix or override the regex used to populate the field? Do I need to specify the field name somewhere to ensure it's updating the field I'd like to update in the props.conf for the specific sourcetype I am trying to manipulate?

 searchHere| rex "(?<longitude_site>(\(\d+\))([A-Za-z0-9_\-]+\(\d+\))+\w+(\(\d+\)))" | rex mode=sed field=longitude_site "s/\(\d+\)/./g s/\.$//g"

[alemarzu]

Just to give you an example on a single instance of Splunk. This should work, had to change your regex a lil bit.

On your props.conf...

[detele_331]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
category = Custom
pulldown_type = true
SEDCMD-longitude = s/(?<=\s)\(\d{1,}\)//g s/\(\d+\)/./g s/\.(?=\s+)//g
EXTRACT-longitude = \w\s{9,}(?<longitude_site>\S+)

From now on after adding this to your props.conf and doing a restart, new events will get modified by your SEDCMD regex.
Then you must add the new field extraction that will be applied to the modified raw data at index time.

Hope it's clear enough now.

[splunker1981]

Here is a sample entry of the data:

02/28/2016 9:07:27 AM 995044 PA  00000000000000 UDP Rcv 192.168.x.x     4959   M [000933   T   NOTRANSMITERROR] L          (35)site_xx(139)tokyo(3)asia-1(0)
  LongResource info at 000000000000xxx
  SocketSend = 356
  SocketRecv = 568

Using the search shown below I am able to convert (35)site_xx(139)tokyo(3)asia-1(0) to site_xx.tokyo.asia01. Currently the value (35)site_xx(139)tokyo(3)asia-1(0) is being extracted into it's own field, but in the format shown, which I'd like to update.

searchHere| rex "(?<longitude_site>(\(\d+\))([A-Za-z0-9_\-]+\(\d+\))+\w+(\(\d+\)))" | rex mode=sed field=longitude_site "s/\(\d+\)/./g s/\.$//g"

thanks for the help.

[alemarzu]

Hi there,

• Can you paste your props.conf ?
• Did you check field extraction permissions ?
• What about your environment, single instance/distributed/cluster ?

[somesoni2]

Few things to understand first
The SEDCMD only applies at index-time. so,
x) If you configure props.conf with SEDCMD on Search Head, it will not have any impact on the indexed data/field extraction
y) If you configure props.conf with SEDCMD on Indexer/Heavy Forwarder (where it should be), it will only update the indexed data which will get ingested after the configuration. Already indexed data will not get changed.
z) Since your SEDCMD is based on an extracted field, it will not be applied, even if you configure it at indexer/HF.

The method that I'll choose will be this
a) Configure SEDCMD at Indexer (will be applied on new data only) to do this replacement based on _raw field (need to update your current regex)
b) Configure field extraction at Search Head to extract your longitude_site field from updated _raw.

Update

Try this in props.conf on the Indexer

[YourSourceType]
...other settings...
SEDCMD-fixsite=s/\s\(\d+\)/site="/
SEDCMD-fixsite2=s/\(\d+\)/./g
SEDCMD-fixsite3=s/\.\s/"/

Current events:

02/28/2016 9:07:27 AM 995044 PA  00000000000000 UDP Rcv 192.168.x.x     4959   M [000933   T   NOTRANSMITERROR] L          (35)site_xx(139)tokyo(3)asia-1(0)
   LongResource info at 000000000000xxx
   SocketSend = 356
   SocketRecv = 568

After this props config

02/28/2016 9:07:27 AM 995044 PA  00000000000000 UDP Rcv 192.168.x.x     4959   M [000933   T   NOTRANSMITERROR] L         site="site_xx.tokyo.asia-1"   LongResource info at 000000000000xxx
   SocketSend = 356
   SocketRecv = 568

[asimagu]

any chance you could share where the extraction needs to take place?

[splunker1981]

Sorry, not sure I follow what you're asking for? Do you just need a sample of what the longitude_site looks like?

[alemarzu]

He is asking for a data sample mate, that would be helpful.