How to edit my props.conf and transforms.conf to extract field names and values from my events?

[edrivera3]

Hi

I want to extract the field names and field values of my events.
My event looks like this:

Step:  1000
Result: blabkbk
Actual: blabblabl
Step:  1100
Result: blabkbk
Actual: blabblabl 

I want the field name to be "1000" and "1100"and the respective field values to be everything below them. This is how I set up my props.conf and transforms.conf, but I am not extracting anything. I appreciate your help.

props.conf

[<mysourcetype>]
REPORT-step_num = step_num

transforms.conf

[step_num]
REGEX = STEP:\s+(?<_KEY_1>/d+)\n(?<_VAL_1>[\w\W\n]+?)\nSTEP

[woodcock · Accepted solution]

Like this in transforms.conf:

REGEX=(?ms)^Step:\s*([^\r\n]+)[\r\n]+(.*?)(?=\Z|[\r\n]+Step:)
CLEAN_KEYS = false
FORMAT = $1::$2
MV_ADD = 1

You should be able to test it on directly in search like this:

... | rex max_match=0 "(?ms)^Step:\s*(?<_KEY_1>[^\r\n]+)[\r\n]+(?<_VAL_1>.*?)(?=\Z|[\r\n]+Step:)"

[edrivera3]

I tried the regex and it doesn't extract anything. Also I read that the name of the group must start with an alpha value.

I tried the following first regex inline and it worked. I tried the second one and Splunk displayed an error related with the alpha value issue.

... | rex "STEP:\s+\d+\n(?<myvalue>[\w\W\n]+?)\nSTEP" 
... | rex "STEP:\s+\d+\n(?<1myvalue>[\w\W\n]+?)\nSTEP" 

[woodcock]

The _KEY_1 and _VAL_1 are special (valid) exceptions:

http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/Configureindex-timefieldextraction#Add_a_rege...

I was guessing that it could be tested inline but perhaps it cannot. In any case, the transforms.conf will work. The (ms) part is critical.

[edrivera3]

Yes, but I was talking about the invalid field names which are "1000" and "1100" in my example.

[woodcock]

Those names are valid for Search Time, provided that you tell splunk not to clean them. To do that, be sure to set this:

CLEAN_KEYS = false

http://www.splunk.com/base/Documentation/6.3.0/Admin/Transformsconf

• NOTE: This attribute is only valid for search-time field extractions.
• Optional. Controls whether Splunk "cleans" the keys (field names) it extracts at search time. "Key cleaning" is the practice of replacing any non-alphanumeric characters (characters other than those falling between the a-z, A-Z, or 0-9 ranges) in field names with underscores, as well as the stripping of leading underscores and 0-9 characters from field names.
• Add CLEAN_KEYS = false to your transform if you need to extract field names that include non-alphanumeric characters, or which begin with underscores or 0-9 characters.
• Defaults to true.

[sundareshr]

How have you configured you LINE-BREAKER? One suggestion is to set your LINE-BREAKER as follows

SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = Step:

Then you can define you AUTO-EXTRACT to be something like this

REGEX = ([^\W]+):=\s+([^\W]+
FORMAT = $1::$2

Just a thought. Not sure I understand your questions. What do you expect your end result to look like? How have you set up your event line-breaking? From your example, would you like Step: 1000 Result: blabkbk Actual: blabbk to be one event and so on?

If that's how you would like to see the events, then try adding the following to your props.conf

SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = Step:

And for field extraction (AUTO-EXTRACT) you could use something like

REGEX = ([^\W]+):=\s+([^\W]+) 
FORMAT = $1::$2

The above is just a general idea to get you started. Hopefully this helps.

[edrivera3]

The event cannot be divided because there are relevant information at the beginning of the event related to all the steps. If I divided the steps of the event, then the information in this step would not have any reference of what's about and they would be useless.

[Richfez]

Just a note: STEP does not equal Step.