Getting Data In

How to send lines to the nullQueue *before* applying line breaking

splunk_zen
Builder

An app of ours spits such a huge volume of data when our Devs increase its debug level to Trace that it essentially results on the Splunk Heavy forwarders sitting on those servers to stall and becoming non operational
(we implemented a script to auto restart them when we don't have indexed data on those servers for more than 10m)

As far as I know it's only possible to send events to nullQueuing, ie, after properties like line breaking (which is costly but vital for us for those logs) have already been computed.

Any way we can setup props / transforms / anything to first completely ignore the TRACE lines and only then applying the props to the logs?

0 Karma
1 Solution

jeffland
SplunkTrust
SplunkTrust

No, you can't use nullqueue before linebreaking - because only after your data is broken into lines can you throw lines away/into the nullqueue. Before that, you could of course apply a regex to determine where to route stuff, but you would then apply those settings to various lines and events (for example if you're reading a file, those are read in chunks of 64k - would you like to apply the routing to that chunk?)
If you desire deeper insight, I would recommend the talk "How splunkd works" from .conf2014, check it out here. It details how the different queues are applied in order and how they work together pretty well.

View solution in original post

jeffland
SplunkTrust
SplunkTrust

No, you can't use nullqueue before linebreaking - because only after your data is broken into lines can you throw lines away/into the nullqueue. Before that, you could of course apply a regex to determine where to route stuff, but you would then apply those settings to various lines and events (for example if you're reading a file, those are read in chunks of 64k - would you like to apply the routing to that chunk?)
If you desire deeper insight, I would recommend the talk "How splunkd works" from .conf2014, check it out here. It details how the different queues are applied in order and how they work together pretty well.

splunk_zen
Builder

Thanks Jeff, that was my initial assumption.

0 Karma

jeffland
SplunkTrust
SplunkTrust

Assuming that the diagram on this page is still up to date, then the nullqueue comes into effect after parsing and aggregation queue during typing queue. I don't think it's possible to change this within splunk.

splunk_zen
Builder

Yeah, was aware of the Splunk data flow, was just curious if someone had implemented something which circumvented this. Thank you anyway Jeff

0 Karma

merp96
Path Finder

Hi,

You could setup a props and transforms to channel such events to null queue

props.conf
[source::/.../debug]
TRANSFORMS-filterdata=set_nullqueue_debug

transforms.conf
[set_nullqueue_debug]
REGEX=(debug)
DEST_KEY=queue
FORMAT=nullQueue

You would need to create a regex which matches all the debug events from source debug and channel them to nullqueue. All other events will get indexed.

0 Karma

splunk_zen
Builder

Thanks but that was our first approach, which didn't answer my requirement of dropping those events before the line breaking. The reason why this was so important was this app function logging was poorly written and was generating one line with millions of characters

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...