Getting Data In

How to configure proper line breaking in props.conf on the universal forwarder for my sample data?

horsefez
SplunkTrust
SplunkTrust

Hi beloved Splunkers,

I'm currently trying to set up a data connection between one of our servers and my Splunk deployment. Unfortunately, I encountered some problems when it comes to Splunk recognizing line-endings and -beginnings.
Let's take a closer look at my current problem.

I have a data file with events that look kinda like that:

<666> this, is, the, event, number, 1,
<666> this, is, the, event, number, 2,
<666> this, is, the, event, number, 3, but, it, is, slightly, longer, than, the, others,
<666> this, is, the, event, number, 4,
<666> splunk, fast, like, a, f-18, bro,
<666> this, is, the, event, number, 6,

What you can see here is, that all those events have something in common.
Yeah, its the "< 666 >" part.

Splunk is flawless I give you that, but for some reason, it sometimes combines two single events into one.

So I was thinking that I need to configure a stanza in props.conf on the forwarder to tell splunk how to deceide when a new event starts.
I did write one, but failed.... maybe?!?

[source::/path/to/file/]
BREAK_ONLY_BEFORE = (\<\d+\>)
SHOULD_LINEMERGE = True

I would love to know if someone out there is brave enough to tell me the right solution.
Thank you for your help, bro/sis!

Regards,
pyro_wood


Splunk> like a F-18, bro ♥

0 Karma
1 Solution

skoelpin
SplunkTrust
SplunkTrust

You have the correct approach, but you need to edit your props.conf on the indexer rather than the forwarder. Don't forget to restart your Splunkd Service after making these changes!

View solution in original post

skoelpin
SplunkTrust
SplunkTrust

You have the correct approach, but you need to edit your props.conf on the indexer rather than the forwarder. Don't forget to restart your Splunkd Service after making these changes!

horsefez
SplunkTrust
SplunkTrust

Quick answer. I like it. I'll try.
Thanks skoelpin-bro!

0 Karma

horsefez
SplunkTrust
SplunkTrust

Did that, but there are still events that get combined into one event.
What statement for props.conf would you choose if you'd had events like mine.

Could you provide me with an example maybe?
Thank you!

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Do the events have the same timestamp? Try hardcoding the digits in your regular expression to see if that will work. If the below stanza doesn't work then I would suggest hardcoding the host instead of the source and try that. Don't forget to restart the Splunk service, these changes won't work until the service is restarted

[SOURCE]
TIME_FORMAT = %Y-%m-%d %H:%M:%S,$3N
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE = \<666\>

horsefez
SplunkTrust
SplunkTrust

Kudos to you my friend!

The suggestion with the time_format did the trick. I didn't realize that the event had two dates in it. One at the start and one further in. Seems like Splunk wasn't able to determine the correct time format before or this messed up the separation of events.

Anyway....

Works!
Big Thanks! 🙂

s2_splunk
Splunk Employee
Splunk Employee

I would slightly modify that to say BREAK_ONLY_BEFORE = ^\<666\> to anchor the string to the beginning of the line unless you can guarantee the string doesn't occur elsewhere. Just a thought.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

It looks like there is a space before <666> which would break the regular expression I provided. If you wanted to use the ^ and anchor this in, you then need to do ^\s\<666\> OR ^\s\<\d{3}\>.. Regex is super easy to learn, I'd recommend using a site like www.regex101.com to test your regex to verify it works

0 Karma

horsefez
SplunkTrust
SplunkTrust

Thank you. I'm relatively new to regular expressions so I always wondered what the ^ is for.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...