Splunk Search

best way to automatically rename fields that have been generically named

sideview
SplunkTrust
SplunkTrust

Let's say you've got a custom application log that has a lot of sensibly named fields. But in addition to the sensible ones, there are a few other fields with generic names like "field1", "field2", "field3", etc...

The semantics of what these fields actually are varies with the value of another field called "event_type_id".

For example, if we have an event with event_type_id="download", the field1 value might be the filename and the field2 value might be the size in bytes. Likewise with event_type_id="client_error", the field1 value might be an error code, and the field2 value might be a description.

I'm looking for the cleanest way to repair this and get these generic fields renamed back to the relevant sensible name.

My question is -- what's the cleanest way to solve this problem and get the field names all renamed automatically in all my searches.

Tags (2)
0 Karma

sgadde
Explorer

Query = ..... | eval zip=mvzip(splunk_column_name, actual_column_name) |mvexpand zip |rex field=zip "(?<splunk_column_name>[^,]+),(?<actual_column_name>[^,]+)" | fields - zip |eval zip=mvzip(splunk_column_name,actual_column_name) |foreach field* [eval x<<MATCHSTR>>=if(match(zip,"^<<FIELD>>"),replace(zip,"^.*,",""),'')] | foreach field* [eval {x<<MATCHSTR>>}=<<FIELD>> | fields - x<<MATCHSTR>>,zip] | stats values(*) by ***[list of fields] *** | rename values(*) AS *

splunk_column_name is a multivalued field containing the list of field names (field1, field2, field3,..)
actual_column_name is a multivalued field containing the list of actual field names
( 1. for event_type_id="download" , you have filename, bytes ...
2. for event_type_id="client_error" , you have error_code, description .... )

| eval zip=mvzip(splunk_column_name, actual_column_name) |mvexpand zip |rex field=zip "(?<splunk_column_name>[^,]+),(?<actual_column_name>[^,]+)" | fields - zip |eval zip=mvzip(splunk_column_name,actual_column_name)

This part of the query expands the multivalued fields splunk_column_name, actual_column_name and creates a new field named zip which contains these the values in these two fields combined.

|foreach field* [eval x<<MATCHSTR>>=if(match(zip,"^<<FIELD>>"),replace(zip,"^.*,",""),'')]

This part of the query creates new fields x1, x2, x3 .... (x1 contains the column names for field1, x2 for field2 ......)
You will get something like this

Timestamp,event_type_id,field1,field2,splunk_field_name,actual_field_name,zip,x1,x2
3/20/16 12:00:00.000 AM,download,file1,30,field1,filename,"field1,filename",filename,
3/20/16 12:00:00.000 AM,download,file1,30,field2,size in bytes,"field2,size in bytes",,size in bytes
3/20/16 12:00:00.000 AM,client_error,404,Not Found,field1,error_code,"field1,error_code",error_code,

3/20/16 12:00:00.000 AM,client_error,404,Not Found,field2,description,"field2,description",,description

| foreach field* [eval {x<<MATCHSTR>>}=<<FIELD>>
Creates new fields with the names of fields as retrieved from x1, x2 ,... and takes value from field1, field2,...

Timestamp,event_type_id,filename,size in bytes,error_code,description
3/20/16 12:00:00.000 AM,download,file1 ,,,

3/20/16 12:00:00.000 AM,download,,30,,
3/20/16 12:00:00.000 AM,client_error,,,404,
3/20/16 12:00:00.000 AM,client_error,,,,Not Found

| fields - x<<MATCHSTR>>,zip] | stats values(*) by ***[list of fields] *** | rename values(*) AS *
Removes x1, x2, x3 .... , zip and use stats to get values for all fields for a particular event in one row.

Timestamp,event_type_id,filename,size in bytes,error_code,description
3/20/16 12:00:00.000 AM,download,file1,30,,
3/20/16 12:00:00.000 AM,client_error,,,404,NotFound

0 Karma

sideview
SplunkTrust
SplunkTrust

One way that I know of is to chain together a lot of conditional eval statements.

| eval filename=if(event_type_id="download",field1,filename) 
| eval bytes=if(event_type_id="download", field2, bytes)
| ....

However I'm curious if I'm overlooking some unusual way of doing it with lookups, or a way to do it with vastly less eval statements.

Like you could maybe have a lookup from eventtype_id="download" to a value like "field1:filename,field2:bytes", and then do some eval magic, possibly involving things like eval {newFieldName}=oldFieldName.

I'm still open to accepting new answers - wild and crazy ones or sensible ones.

0 Karma

gelica
Communicator

Did you find a way to do the renaming with a lookup?
I have a similar problem, where I want to rename a field and then automatically show the new field in my searchresult.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...