Getting Data In

line breaking partial success

jsmith_splunk
Splunk Employee
Splunk Employee

I'm trying to bring in some custom source log files and initially no line breaking was occurring so all of the events in each log file would arrive as 1 event.
I wrote a line break based on each new record starting with a 7 digit date string.

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\d{7}\s+

Once that took effect, any log files that strictly contained only contained lines starting with a 7 digit string parsed correctly. However any log files that anywhere in the file has a line not starting with a 7 digit string, the entire log file is still entered as 1 event. As all lines not starting with the 7 digit string should be treated as the data for the previous digit string I'm unsure why this is failing.

0 Karma
1 Solution

jsmith_splunk
Splunk Employee
Splunk Employee

Thanks all, I ended up resolving the issue by writing the following in props.conf

[sourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ^\d{7}

View solution in original post

0 Karma

jsmith_splunk
Splunk Employee
Splunk Employee

Thanks all, I ended up resolving the issue by writing the following in props.conf

[sourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ^\d{7}
0 Karma

lguinn2
Legend

If every line is a single event, you can do this very simply:

SHOULD_LINEMERGE = false

That is all that is needed. Do not set the LINE_BREAKER if every line is a single event.

If your input has timestamps that are not always at the beginning of a line, then (as @woodcock points out), you need to parse out the timestamp. There are a variety of settings for this. For multi-line events, Splunk breaks to begin each event with a line containing a timestamp. If Splunk isn't parsing your timestamps properly, it can't break multi-line events properly. And even if your input is all single-line events, you do need to get the timestamp right, too!

As @somesoni2 mentioned, samples of your data will help us help you.

Finally - if you are collecting multiple log files with a single input stanza, that's great. But if the log files have different formats, you will need to treat them differently for parsing by using separate props.conf stanzas.

0 Karma

woodcock
Esteemed Legend

Generally the solution is to set your timestamp and break there. Do your events start with a timestamp? Share a few sample events.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Could you post some sample data , including both type of events, one starts with 7 digit string and one without it,

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...