Splunk Search

Mapping Fields to Meaningful Names

cgl
Explorer

I have a vendor log file that has numeric codes for the field names (i.e. E-1, E-710, etc). The vendor also provides a lookup table that translates the codes into meaningful names (i.e. Asset Name, CPU Time, etc).

I am currently adding the translation via FIELDALIAS in the props.conf, but I'd rather provide the translation via a lookup table for convenience and maintainability. Also I'd prefer to replace the numeric code with the meaningful name rather than create duplicates.

Any ideas on how I should proceed?

Thanks in advance,

Chris

Tags (3)

hexx
Splunk Employee
Splunk Employee

This should be possible if your key/value pairs are present in the raw data of your events.

Let's take a hypothetical example, assuming that this is the raw data of one of your events :

20 Jan 2011 19:34:22 E-1=100 Purgamentum init, exit purgamentum

Let's say that field name "E-1" needs to be translated to "Error message" and field value "100" needs to be translated to "Banjo infestation".

The first thing we want to do is to extract the "E-1" and "100" as a field each, not as a key/value pair. Let's do this with a rex :

| rex "\d2:\d2:\d2 (?<field_name_code>[^\=]*)\=(?<field_value_code>[^\s]*)\s

So now we have two new fields : field_name_code = E-1 and field_value_code = 100.

This is where your lookup comes in (let's say it's defined in transforms.conf as "errorlookup"] to output two new fields : "field_name" which will be looked up based on "field_name_code" and "field_value" which will be looked up based on "field_value_code".

| lookup errorlookup field_name_code field_value_code OUTPUT field_name field_value

So we'll now have a new field named "field_name" with a value of "Error message" and another one called "field_value" with a value of "Banjo infestation".

Now you want to create a new field that concatenates "field_name" and "field_value" with a known delimiter (let's say "=") :

| eval error_kv=field_name."=".field_value

We'll now have a new field called "error_kv" with a value of "Error message=Banjo infestation".

The final step is to use a delimiter-based search-time field extraction scoped to the "error_kv" field in order to create a new field named "field_name" with a value of "field_value". The options of the kv command do not allow for this to be done inline, so we'll need to define our field extraction in $SPLUNK_HOME/etc/system/local/transforms.conf (or anywhere else that is adequate to your configuration) :

[extract_error_kv]
SOURCE_KEY = error_kv
DELIMS = "="

And now, we can call this custom extraction inline using kv :

| kv extract_error_kv

In our example, this should result in the creation of the field "Error message" with a value of "Banjo infestation"!

Part of this is theory, so I'm quite curious to know if it works in your environment. Please let me know!

piebob
Splunk Employee
Splunk Employee

+1 for banjo infestation

cgl
Explorer

It's okay, I appreciate you taking the time to respond.

Ayn
Legend

Ahh, I see - you're talking about the names of the fields themselves, not their values? I don't have any better solution than the one you're already going with, sorry...

cgl
Explorer

The lookup is a CSV, but the lookup only maps the values, not the fields. For example,

E-1=100, could use the lookup table to map to E-1=Error Message

However, I haven't been able to map the E-1 to a better presentation value, expect via the FIELDALIAS option.

Ayn
Legend

Seeing as you're provided with a lookup table from the vendor, you should absolutely be using that, especially if it's in CSV format - just put it in the lookups directory, configure the lookup in props.conf / transforms.conf, and you're all set!

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...