Getting Data In

send to nullqueue events which have more than 100 lines

robertosegantin
Path Finder

I have an XML file which has events made by many rows.
I would like to send to null queue the events which have more than 100 rows.
How can I do that?

0 Karma
1 Solution

DavidHourani
Super Champion

Hi @robertosegantin,

I see three ways to go about this:

1- The easiest way is to set TRUNCATE limit in props.conf for your sourcetype in order to avoid having more than a fixed number of a characters per event. This however will not completely remove the event but merely filter part of it.

2- (Recommended) Use a combination of regexp to identify the long events and move them to the nullqueue:
https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_...

3- Use a regular expression to select the useful info from the long 100 line event, and discard the rest. You can find this here:
https://answers.splunk.com/answers/735219/index-selected-lines-in-a-multiline-event.html

I think 2 is best suited for what you're trying to achieve. You can use a regex to count the number of lines if needed, something like : (.*(\n|\r)){100}, then send to null queue anything that matches.

Let me know if that helps.

Cheers,
David

View solution in original post

0 Karma

DavidHourani
Super Champion

Hi @robertosegantin,

I see three ways to go about this:

1- The easiest way is to set TRUNCATE limit in props.conf for your sourcetype in order to avoid having more than a fixed number of a characters per event. This however will not completely remove the event but merely filter part of it.

2- (Recommended) Use a combination of regexp to identify the long events and move them to the nullqueue:
https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_...

3- Use a regular expression to select the useful info from the long 100 line event, and discard the rest. You can find this here:
https://answers.splunk.com/answers/735219/index-selected-lines-in-a-multiline-event.html

I think 2 is best suited for what you're trying to achieve. You can use a regex to count the number of lines if needed, something like : (.*(\n|\r)){100}, then send to null queue anything that matches.

Let me know if that helps.

Cheers,
David

0 Karma

robertosegantin
Path Finder

Hi @DavidHourani ,

thanks for your answer.
I tried:

== props.conf ==
[my]
CHARSET = UTF-8
KV_MODE = xml
MAX_EVENTS = 100000
MAX_TIMESTAMP_LOOKAHEAD = 28
NO_BINARY_CHECK = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%4Q
TIME_PREFIX = ^[
category = Custom
disabled = false
pulldown_type = true
TRANSFORMS-nullqueue_more_than_100_lines = nullqueue_more_than_100_lines

== transforms.conf ==
[nullqueue_more_than_100_lines]
REGEX = (.*(\n|\r)){100,}
DEST_KEY = queue
FORMAT = nullQueue

But it does not send events with more than 100 lines to nullqueue.
I also tried with TRUNCATE=5000, but is does not work, too

Cheers,
Roberto

0 Karma

DavidHourani
Super Champion

Hi @robertosegantin, Is this working now ?

0 Karma

robertosegantin
Path Finder

Hi @DavidHourani,
sorry for late replay.
The problem is that "TRUNCATE" and "TRANSFORMS" seem to work on single line, and they work before "SHOULD_LINEMERGE", which is true by default.
In this way, when Splunk merges 150 xml lines into one event, the "TRUNCATE" and "TRANSFORMS" option work on single line, which has less than 100 lines (is only one!) and is less than 5000 byte

Have I done some mistake?

Thanks!

0 Karma

DavidHourani
Super Champion

Hi @robertosegantin,

Go for SHOULD_LINEMERGE = false and set a new line breaker which is the closing stanza for your xml. It should do the trick.

0 Karma

robertosegantin
Path Finder

Hi @DavidHourani,

found the golden path! 😄

[my_sourcetype]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=45
disabled=false
LINE_BREAKER =(<\/s:Envelope>)
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%4Q
TIME_PREFIX=(<\/s:Envelope>[\r\n]+)?[
TRUNCATE=3000

Thanks for your help!

0 Karma

DavidHourani
Super Champion

You're welcome!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...