Splunk Search

Why do the contents of the first capturing group in this LINE_BREAKER regex appear as a separate event?

Graham_Hanningt
Builder

I have defined a TCP input in inputs.conf with the following corresponding stanza in props.conf (Splunk Enterprise 6.4):

[source::tcp:6067]
KV_MODE = json
LINE_BREAKER = ((^[^{][^\r]*\r\n)*)\{\"[^}]+\}
SHOULD_LINEMERGE = false

If I send the following text to that port:

Preamble lines
That I do not want
To appear in the event
The following line is intentionally blank

{"myfield": "some_value"}

(with \r\n at the end of each line)

I get two events in Splunk:

  • The event I want, {"myfield": "some_value"}, with myfield correctly presented as a field (so, KV_MODE = json is working).
  • An unwanted event, with a time stamp that is the same or earlier, consisting of the "preamble" lines that I thought I'd told LINE_BREAKER to discard!

According to the props.conf documentation:

The contents of the first capturing group are discarded, and will not be present in any event.

Yes, the contents of the first capturing group are discarded from the event I want... but they are present in that unwanted (and unexpected) event.

Why do I get that unwanted event? How do I prevent it?

I'm deliberately using the descriptive term "preamble" here, because I have previously attempted to do the same thing (discard those "preamble" lines) using PREAMBLE_REGEX instead of LINE_BREAKER:

[source::tcp:6067]
KV_MODE = json
HEADER_FIELD_LINE_NUMBER = 1
PREAMBLE_REGEX = ^[^{].*

but I cannot get PREAMBLE_REGEX to work, no matter what combination of regex and preamble test cases I use; at least, not for a TCP input. I wonder whether PREAMBLE_REGEX only applies to, say, file inputs, not TCP (or other network) inputs. The props.conf documentation hints at this with the word "files":

Some files contain preamble lines.

but if it's true, I'd prefer that the documentation was more explicit (and this makes me wonder about the implicit limitations of other settings).

0 Karma

woodcock
Esteemed Legend

Try adding BREAK_ONLY_BEFORE to make your LINE_BREAKER less aggressive.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...