Splunk Search

Regex set up

TheWzrdOz
New Member

Very much a noob here. I've read (or tried to read!) the docs, I've watched the videos and still it's not doing what I thought it should.

I have loaded my data into Splunk (an XML file) and did a rex on the data to find what I was looking for. That worked fine.

sourcetype="xml" | rex "<CommandId>(?<commandid>d+)</CommandId>"

Now I want to persist the data. I added this to my props.conf file (in Splunk\etc\system\local):

[xml]
REPORT-uploads = commandId

And this to transforms.conf (same location):

[commandId]
REGEX=<CommandId>(?<commandId>d+)</CommandId>

Did I do this correctly?

Tags (4)
0 Karma

lguinn2
Legend

Second answer:

If you want a direct answer to your question 🙂

I think your regex is probably correct, although I would escape the < characters in the regex (making them \< ) because < by itself has a special meaning to regex.

So if you want to continue with your original solution, try that and forget the xmlkv
But your syntax is a bit wrong, too, I think you need the following in transforms.conf

[commandId]
REGEX=\<CommandId>(d+)\</CommandId>
FORMAT=commandid::$1

Yeah, the markdown gets screwy sometimes when you put in XML or HTML stuff...

0 Karma

lguinn2
Legend

So first, nicely done so far - but I think you are doing too much work!

With Splunk, you can use the xmlkv command and ask Splunk to do the parsing that you are doing by hand. The only issue is that xml parsing can be slow, so you should search first and then parse. For example, if you are looking for commandid=xyz27, run this search string

sourcetype=xml CommandId xyz27 | 
xmlkv | 
search CommandId=xyz27 | 
whateveryouwant

How this works:

  • The first line searches for any events that have both the terms "CommandId" and "xyz27". This is not the same as searching for CommandId=xyz27 but it will eliminate a lot of events. This means fewer events to parse in the next step
  • The xmlkv command will parse the remaining events. Anywhere it sees <tag>value</tag>, it will create a field named tag and set its value to "value". Voila, fields extracted!
  • The next step actually tests for CommandId=xyz27, eliminating any events that do not match
  • Finally, you get to do whatever you were ultimately trying to do (statistics, timechart, etc)

If you are going to do this a lot, you might consider saving it as a macro, once you get it sorted out.

TheWzrdOz
New Member

[xml]

REPORT-uploads = commandId

[commandId]
REGEX = <CommandId>(?\d+)</CommandId>
FORMAT = commandId::$1

0 Karma

TheWzrdOz
New Member

sourcetype="xml" | rex "<CommandId>(?\d+)</CommandId>"

0 Karma

TheWzrdOz
New Member

Markdown is really screwing with the formatting of the text. Not certain how to get around it, either!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...