Getting Data In

Index selection conditional on values in the data

davidatpinger
Path Finder

I've got a bunch of key-value data, something sorta like this:

a=1,b=2,c=3,d=4
a=5,b=6,c=7,d=8
a=9,b=2,c=10,d=11
(etc.)

I'd like to sort this data into different indexes (for the purpose of different retention times) depending on the value passed to the 'b' key. So, if b=2, send the data to index_retain_for_one_week but if b=6, send the data to index_retain_for_one_month. Ideally, there would be a final condition for values of 'b' that aren't listed. Think of this as an 'else' condition that sends non-matching data for the list of conditions to index_retain_one_day. (All of the index names are just illustrative, like the data.)

Is there a good way to do that? Heck, is it possible? My apologies if this is already answered somewhere - I couldn't find a set of key words that generated an answer. Thanks!

0 Karma
1 Solution

davidatpinger
Path Finder

I suppose I can make multiple stanzas of transforms and they are applied in order by the TRANSFORMS statement in props.conf. So yeah, this will work! Thanks!

0 Karma

davidatpinger
Path Finder

Okay, this is close but not quite working. I've got something like this in transforms.conf:

[special-data]
DEST_KEY = _MetaData:Index
REGEX = b=[3|30|44|49|21]
FORMAT = special-index

[normal-data]
DEST_KEY=_MetaData:Index
FORMAT = normal-index

And then, in props.conf, I have something like this:

[mysourcetype]
TRANSFORMS-indexsort = special-data, normal-data

There must be something unhappy with the REGEX, because everything ends up in special-index. Hmm.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You forgot to add "REGEX = ." in the normal-data stanza.

[special-data]
DEST_KEY = _MetaData:Index
REGEX = b=[3|30|44|49|21]
FORMAT = special-index

[normal-data]
REGEX = .
DEST_KEY=_MetaData:Index
FORMAT = normal-index
0 Karma

davidatpinger
Path Finder

Yeah, and I need parens instead of square brackets. Getting there! (Thanks!!)

0 Karma

davidatpinger
Path Finder

Hmm, now everything is falling through to normal-data. Time to muck around with it some more.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Check the REGEX for special-data, may be some spaces that need to be adjusted etc. If you can send some actual sample data, I can try to look at it as well..

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Also, can you do this, in your porps.conf and transforms.conf, change the order of the stanza, so get the normal-data first and special-data after that.

0 Karma

davidatpinger
Path Finder

I discovered that my brain was backwards. It's not first-match and stop in the listed transforms in props.conf - it runs to the end and the last match is what you get.

Once I got that through my skull, everything works as expected. Thanks much!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...