dynamic rex pattern

[klee310]

I'm trying to use rex to extract a value from an event.

In order to avoid writing out the pattern too many times, I have decided to place the pattern inside a macro with a specified argument passed in.

First, sample data looks like this:

DataType=1, PowerMax=50, PowerMin=10

To invoke the macro, I might use a command like this:

`macFoo("PowerMax")`

and inside the macro, I want to do something like this:

... | eval re=", ".$arg1$."=(?<bar>[^,]*)" | rex field=_raw re

which i wanted to extract bar=50, but I get an error similar to this:

Error in 'rex' command: The regex 're' does not extract anything. It should specify at least one named group. Format: (?<name>...).

Help!

[Ayn]

Yeah while I provided an answer to the specific question you had below, I agree that you might be onto the wrong path here. That is often the case when you have to resort to the kind of ugliness that's in my answer 🙂

[klee310]

somesoni2, thanks for the reply. Yes those kvp's are automatically extracted by Splunk, but as related to my other question (http://answers.splunk.com/answers/114240/dynamic-field-value-extraction), I can't use those field directly unless I hard-code the field I want to extract/compare

[Ayn · Accepted solution]

You can use subsearches to achieve this. It's a bit ugly but does the job. We're going to be using that subsearches treat the fields "query" and "search" differently than other field names in the way that the field names aren't used in the output. So

[... | eval foo="bar" | fields foo]

would return something like

((foo="bar"))

whereas

[... | eval query="bar" | fields query]

would return

(("bar"))

We can't use this output right away in your scenario though because of the parantheses. Thankfully you can change the format that's used by the subsearch when returning results, by invoking the command format with the proper parameters at the end. In this case we just want to remove all parantheses so we just set empty strings for everything:

[... | eval query="bar" | fields query | format "" "" "" "" "" ""]

This will return

"bar"

We can now use this in your regex case. (The stats count at the beginning of the subsearch is just a dummy search, it's just there to be able to run the eval)

... | rex field=_raw [|stats count | eval query=", ".$arg1$."=(?<bar>[^,]*)" | fields query | format "" "" "" "" "" ""]

[klee310]

Further to this, is there any other way to extract the value of "bar", if I had .. | eval foo="bar" ?

[klee310]

YES. This works, but only to a certain extent. It seems the engine doesn't like to see the comma inside the character class, ie [^,]... the error I get is the following... Error in 'rex' command: Encountered the following error while compiling the regex 'PowerMax=(?[^': Regex: missing terminating ] for character class... the fix I can work with at the moment is to use \d+ character-class, but I will eventually still need a way to catch everything before a comma. Any ideas?

[klee310]

awesome! as ugly as it might be, I psyched to give it a go. Will respond after I have had a try at it...

[somesoni2]

Just one doubt, your sample data pretty much looks like ideal input data for splunk (key value pair which are , separated) and Splunk should already have extracted all these field. You sure fields are not automatically extracted and you need a rex to do it?

[klee310]

thanks Ayn for the quick response. I can gather as much from the error message. Any ideas as to how I can achieve this by other means?

[Ayn]

No answer/solution to your problem but at least an explanation, so just putting this in as a comment: rex does not interpret "re" as the variable you just created. It will interpret it as the STRING "re", which is why it will complain that you're not extracting anything.