Splunk Search

How do I create key/value pairs from a _raw field with only values?

joshua_hart
Explorer

I have a Symantec Messaging Gateway syslog input that provides syslog with no keys, only values. For example:

2013-07-11T13:13:16-04:00 appliance-name ecelerity: 1373562795|d6038c16-b7fe96d000000710-2d-51dee7aae3dd|SENDER|some-email-address@domain.tld

Right now this entire event is contained within the "_raw" field. The log data is everything after "ecelerity:" and each value is delimited by a pipe character. What I'd like to do is create fields for those values and then index the event so I can search on those fields. The example above would have five fields:

  • Epoch Time
  • Unique ID
  • Action
  • Sender Address

It's important to note that the above example is but one among many. Some of the other events have more values and the keys for those values will differ based on the type of event (though everything up to and including the 'Action' field would be consistent across events).

What I need is the means to parse these events and then create rules for each event to add keys to the values. How can I do this? I'm thinking something in the props/transforms, but I'm not exactly sure how.

0 Karma
1 Solution

sdaniels
Splunk Employee
Splunk Employee

You could start with the Splunk Interactive Field Extractor (IFX) to parse out your fields for you. It will generate the appropriate regex for you. Sometimes it may need to be tweaked though.

http://www.splunk.com/view/SP-CAAADUY
http://docs.splunk.com/Documentation/Splunk/5.0.3/Knowledge/ExtractfieldsinteractivelywithIFX

And yes you are correct that you'll use props.conf and transforms.conf to manually extract out fields. The IFX will write out data to those config files so you'll see the examples it creates. You should see those additions under $SPLUNK_HOME/etc/users.

http://docs.splunk.com/Documentation/Splunk/5.0.3/Knowledge/Addfieldsatsearchtime

There is also a way to extract fields on the fly in a search if it's something less common and you don't already have a field:

http://docs.splunk.com/Documentation/Splunk/5.0.3/Search/Extractfieldswithsearchcommands

View solution in original post

0 Karma

the_wolverine
Champion

If your data originates from a file that contains a header, I would use automatic header-based fields: http://docs.splunk.com/Documentation/Splunk/5.0.3/Data/Extractfieldsfromfileheadersatindextime

0 Karma

sdaniels
Splunk Employee
Splunk Employee

You could start with the Splunk Interactive Field Extractor (IFX) to parse out your fields for you. It will generate the appropriate regex for you. Sometimes it may need to be tweaked though.

http://www.splunk.com/view/SP-CAAADUY
http://docs.splunk.com/Documentation/Splunk/5.0.3/Knowledge/ExtractfieldsinteractivelywithIFX

And yes you are correct that you'll use props.conf and transforms.conf to manually extract out fields. The IFX will write out data to those config files so you'll see the examples it creates. You should see those additions under $SPLUNK_HOME/etc/users.

http://docs.splunk.com/Documentation/Splunk/5.0.3/Knowledge/Addfieldsatsearchtime

There is also a way to extract fields on the fly in a search if it's something less common and you don't already have a field:

http://docs.splunk.com/Documentation/Splunk/5.0.3/Search/Extractfieldswithsearchcommands

0 Karma

joshua_hart
Explorer

Using the IFX seemed to work for now. I wasn't able to extract all the fields I was looking for, but I was able to get at what I needed for our purposes.

Ideally, if I had the option to format the data before being sent to syslog, I'd be happy. In fact, if Symantec didn't send Brightmail mail audit logs to syslog as separate events (each aspect of a single record is sent as a separate syslog event) I'd have a much easier time extracting fields.

Thanks for the tips, BTW.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...