Getting Data In

Can I still send data to nullQueue while using _MetaData:Index to send data to an Alternate Index?

mmcgrew
New Member

I have one source directory in the inputs.conf file that I need to parse out and send different events to different Indexes.
I attempt to do this by using the _Metadata:Index Key within the transforms.conf file. While this works, ( I can direct events to different indexes using _Metadata:Index), I cannot send my unwanted items to nullQueue. The problem this creates is all the leftover events (that didn't match a regex statement I had in transforms.conf) gets sent to the default Index I defined in inputs.conf. I should be able to send all items that don't match my regex statement to nullQueue but I cannot do this while utilizing the _Metadata:Index Key in transforms.conf....looking below at the examples I've provided, once I use the "TRANSFORMS-null=setnull" in the props.conf file - nothing works (none of my data ends up in any index). I have tried moving the "TRANSFORMS-null=setnull" around to the top and bottom of the props file but still get nothing...once I remove the "TRANSFORMS-null=setnull" statement from the props.conf file, I get the data I want in index2 and index3 but then get all the leftover data I don't want in index1. It seems as though whatever index is defined in the inputs.conf file will get all the leftover items as it is interpreted as the default index. If I could use nullQueue, then everything would be good. Any suggestions or help is appreciated. Thanks.

inputs.conf
[monitor:///opt/sdata/events/server-A/splunk*]
host = server-A
source = A
sourcetype = data1
index = index1
disabled = 0

props.conf
[host::server-A]
TRANSFORMS-set1 = setparsingdata1
[host::server-A]
TRANSFORMS-set2 = setparsingdata2
[host::server-A]
TRANSFORMS-null = setnull

transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsingdata1]
REGEX = ((^(?=.Summary="BGP Peer Connection Established.").+))
DEST_KEY = _MetaData:Index
FORMAT = index2

[setparsingdata2]
REGEX = ((^(?=.Summary="BGP Peer Connection Idle.").+))
DEST_KEY = _MetaData:Index
FORMAT = index3

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Give this a try [untested, to try on some sandbox first) (no changes to inputs.conf)

props.conf (make sure to keep the setnull as leftmost entry)

[host::server-A]
TRANSFORMS-set1 = setnull,setparsingdata1,setparsingdata2

Updated
transforms.conf

[setnull]
REGEX = ^(?!.Summary=\"BGP Peer Connection (Established|Idle).)
DEST_KEY = queue
FORMAT = nullQueue

[setparsingdata1]
REGEX = ((^(?=.Summary="BGP Peer Connection Established.").+))
DEST_KEY = _MetaData:Index
FORMAT = index2

[setparsingdata2]
REGEX = ((^(?=.Summary="BGP Peer Connection Idle.").+))
DEST_KEY = _MetaData:Index
FORMAT = index3
0 Karma

mmcgrew
New Member

Thank you for the quick response. I just tried this but no luck...no data getting to any of the indexes. Once I remove the setnull...it works - but then I end up indexing a bunch of unwanted events.

0 Karma

mmcgrew
New Member

I have tried this method that you suggested but it did not work either. It does not seem that the REGEX statement within the setnull stanza is the problem....regardless of what I change the REGEX to I cannot seem to get the data to parse to any index while using the setnull/nullQueue option...which is why it seems like there is a conflict with using nullQueue and _Metadata:Index because either of those work independently of each other but when used together it doesn't work.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try the updated answer.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...