Getting Data In

Field names in lowercase, transforms.conf

gelica
Communicator

Hi!

I have some different sourcetypes defined by me where I'm extracting some of the fields with stanzas in transforms.conf at search time (I'm using REPORT in props.conf). Here is one example of a stanza I'm using:

REGEX=(?im)[\r\n]+([^\r\n]*name)\: ([^\r\n]+)
FORMAT=$1::$2
MV_ADD=true

This extracts the fields I want, but since I extract the field name like this, the field name may be in uppercase, lowercase or a combination.

Creating new stanzas for each field is not an option since I have a lot of fields and most of my stanzas are of the form shown above, where I just define the ending of the field name, to be able to extract most of them.

I wonder if there is any way to "force" the field names to lowercase?

0 Karma
1 Solution

grijhwani
Motivator

If you look at the answer to a slightly different question (Dealing with key/value pairs with inconsistent key case) the solution is, perhaps, to pre-process the log stream at input time to convert to lower case with sed commands.

View solution in original post

0 Karma

ddrillic
Ultra Champion

It does seem right to convert all characters to lower case in transforms.conf, in the spirit of -

[syslog-header-stripper-ts]
REGEX = ^[A-Z][a-z]+\s+\d+\s\d+:\d+:\d+\s(.*)$
FORMAT = $1
DEST_KEY = _raw

0 Karma

marcoscala
Builder

Sorry, but if I'm right, this transforms just STRIPS out the syslog header. It doesn't convert it to lowercase.

0 Karma

grijhwani
Motivator

If you look at the answer to a slightly different question (Dealing with key/value pairs with inconsistent key case) the solution is, perhaps, to pre-process the log stream at input time to convert to lower case with sed commands.

0 Karma

grijhwani
Motivator

So - you got it to work in the end?

0 Karma

gelica
Communicator

I tried that, but didn't get it to work, hence my confused comments on that answer.
I don't really understand the regex (s/\([A-Za-z0-9]*)/\1\L\2\g) either, I haven't found any information about it(I'm used to regexes so it is only this one that confuses me 🙂 )

However, I tried this approach again, by adding it to my default-clause in props.conf, restarting Splunk and index new files, but I can't get it to work.. :S

0 Karma

landen99
Motivator

It is for SEDCMD in props.conf and it looks like the author got it wrong. Part of the string is regex and the other parts are code: s/regex/text/g
"s" means replace and "g" means global

He should have written:

SEDCMD-contacts = s/([A-Za-z]*)/\L\1/g

Keep in mind that this only changes the raw text at index time and not the field names after extraction at search time, but if you are extracting the field name from the raw text then search time extractions will extract all lower case.

0 Karma

Ayn
Legend

My mistake - you're absoutely right, field names are case sensitive. I don't have a good solution right now though, sorry.

0 Karma

gelica
Communicator

The field values are case insensitive, but the field names are not.

If I have a field called Machine_name, and run a search for Machine_name=* I get the events I'm looking for, but if I search fort machine_name=* instead, I get no results..

0 Karma

Ayn
Legend

I guess this is mostly for cosmetic purposes? Because field names are case insensitive.

0 Karma

marcoscala
Builder

Fields names ARE case sensitive!!! Values aren't.

0 Karma

landen99
Motivator

stats and where both care about case, to name a couple. I suggest using Calculated fields at search time.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...