Splunk Search

Subject: Problem with the CLEAN_KEYS property

MatthewTowey
Path Finder

I am having a problem with the CLEAN_KEYS property.

Here is my raw message that gets appended to my log file. Note: the keys are consist of just numeric values.

Raw Message:
8=FIX.4.4,9=125,35=D,56=BLPUS,49=FMRFIBOOM,52=20120502-08:43:02,34=5662,54=1,55=GOOG3,38=15200,44=20,11=Order1,21=2,60=20120502-08:43:02,40=2,10=219,

It appears that the fields do not get recognized as they are all numeric.
We set the CLEAN_KEYS property to false expecting that we Splunk would successfully extract the numeric values (8, 9, 56 etc) as fields. However, it does not do this. Are our expectations wrong?

We have noted that if we change the message to include an character after the numeric value, i.e.
8a=FIX.4.4,9b=125,35c=D,56d=BLPUS,49e=FMRFIBOOM,etc. - splunk will recognize a, b, c as fields

0 Karma
1 Solution

kristian_kolb
Ultra Champion

I just tried something similar to what you did (just a regular REPORT-extraction), and it worked fine. As you can see, the the numeric field is not in the event per se, but deliberately set to a numeric name in the extraction process. Getting the log level from lines like;

02/May/2012 12:22:41:925 [CRITICAL] some text comes here
02/May/2012 13:02:35:257 [ERROR] some text comes here

in props.conf

REPORT-blaha = unclean_99

in transforms.conf

[unclean_99]
CLEAN_KEYS = 0
MV_ADD = 0
REGEX = \s\[(?<99>\w+)

UPDATE:

I don't believe that CLEAN_KEYS should go into props.conf.

See the docs for transforms.conf and field extraction in general

You could also do it inline in the search with extract.


UPDATE AGAIN:

As I said before, you should not put CLEAN_KEYS in props.conf.

Do as above, but for the transforms.conf settings instead do;

[unclean_99]
CLEAN_KEYS = 0
DELIMS = ",", "="

This tells Splunk that key/value pairs are separated from each other by a comma, and that keys are separated from their values by an 'equals' character. The rest should go automatically, but you might get strange results if your events contain commas elsewhere (like in timestamps).

This worked fine for me with the following log file data (though I used semicolons instead of commas since the timestamp has a comma).

2012-05-03 15:55:17,89;  66=21963; 55=590621239; 44=FX.74 
2012-05-03 15:55:17,90;  66=29886; 55=97211346; 44=FX.8411 
2012-05-03 15:55:17,90;  66=27491; 55=3176432533; 44=FX.1282 
2012-05-03 15:55:17,91;  66=31710; 55=74392336; 44=FX.29486 

This gives me fields 66, 55 and 44

Hope this helps,

Kristian

View solution in original post

0 Karma

kristian_kolb
Ultra Champion

I just tried something similar to what you did (just a regular REPORT-extraction), and it worked fine. As you can see, the the numeric field is not in the event per se, but deliberately set to a numeric name in the extraction process. Getting the log level from lines like;

02/May/2012 12:22:41:925 [CRITICAL] some text comes here
02/May/2012 13:02:35:257 [ERROR] some text comes here

in props.conf

REPORT-blaha = unclean_99

in transforms.conf

[unclean_99]
CLEAN_KEYS = 0
MV_ADD = 0
REGEX = \s\[(?<99>\w+)

UPDATE:

I don't believe that CLEAN_KEYS should go into props.conf.

See the docs for transforms.conf and field extraction in general

You could also do it inline in the search with extract.


UPDATE AGAIN:

As I said before, you should not put CLEAN_KEYS in props.conf.

Do as above, but for the transforms.conf settings instead do;

[unclean_99]
CLEAN_KEYS = 0
DELIMS = ",", "="

This tells Splunk that key/value pairs are separated from each other by a comma, and that keys are separated from their values by an 'equals' character. The rest should go automatically, but you might get strange results if your events contain commas elsewhere (like in timestamps).

This worked fine for me with the following log file data (though I used semicolons instead of commas since the timestamp has a comma).

2012-05-03 15:55:17,89;  66=21963; 55=590621239; 44=FX.74 
2012-05-03 15:55:17,90;  66=29886; 55=97211346; 44=FX.8411 
2012-05-03 15:55:17,90;  66=27491; 55=3176432533; 44=FX.1282 
2012-05-03 15:55:17,91;  66=31710; 55=74392336; 44=FX.29486 

This gives me fields 66, 55 and 44

Hope this helps,

Kristian

0 Karma

MatthewTowey
Path Finder

Thanks Kristian We got it working
the stanza in transforms.conf was not been correctly referenced in props.conf !

0 Karma

MatthewTowey
Path Finder

Hi Kristian

Thanks for the update

The inline search command extract worked perfectly, i.e. | extract auto=f, clean_keys=f, pairdelim="(,", kvdelim="=".
Ideally we would want this working in a non-inline mode.

We have tried editing the props.conf in the $SPLUNK_HOME/etc/system/local folder

E.g.
[source::/]
CLEAN_KEYS = false

But this does not result in the numeric keys being recognized.

Just wondering if you can see anything that jumps out which would indicate why the CLEAN_KEYS is not working

Thanks
Matt

0 Karma

kristian_kolb
Ultra Champion

see update above. /k

0 Karma

MatthewTowey
Path Finder

Hi Kristan

Thanks for your response.

In our case we want CLEAN_KEYS disabled for all the keys in our event. In your example you appear to be setting CLEAN_KEYS to false for a particular name/value pairing in the event.

We have tried setting CLEAN_KEYS=false in $SPLUNK_HOME/etc/system/local/props.conf but this does not work, i.e. numeric fields are not recognized as fields, e.g. 8,9,35 from my original example

Please and Thank you for any future help
Mat

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...