Getting Data In

Splunk Events getting truncated/choppedoff at the begining

vijayakumarkb
Explorer

My Event logs in splunk are getting truncated in the front part.

Is it possible to spllit lines based on below condition.

[logLevel=ERROR] - 2019-03-22 08:00:04,697 +0000 --

log level can be ERROR OR INFO, either one will come in logs.

how to use LINE_BREAKER for this. I tired couple of examples from the posts, but it is not working.

Currently using as before, but not working, event are chopped of from the front.

SHOULD_LINEMERGE=false
TRUNCATE=5000000

Any help please!!

0 Karma
1 Solution

nickhills
Ultra Champion

Use this in your props.conf

SHOULD_LINEMERGE=false
LINE_BREAKER=(^)\[logLevel
TIME_PREFIX=^\[logLevel=\w+\]\s-\s
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%N3
TRUNCATE=10000 #(or remove this as it's the default)
If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

nickhills
Ultra Champion

Use this in your props.conf

SHOULD_LINEMERGE=false
LINE_BREAKER=(^)\[logLevel
TIME_PREFIX=^\[logLevel=\w+\]\s-\s
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%N3
TRUNCATE=10000 #(or remove this as it's the default)
If my comment helps, please give it a thumbs up!
0 Karma

vijayakumarkb
Explorer

Thanks for the Answer Nick..
I added the solun, but it got all the events clubbed into one event.

for Eg :

[logLevel=ERROR] - 2019-03-25 01:39:24,980 +0000 -- "ConsumerService-9-ConnectionPool-Thread-24" com.DeliveryHandler -- message="INFO: First Time" timestamp="1553477964923"
[logLevel=ERROR] - 2019-03-25 01:39:52,094 +0000 -- "ConsumerService-9-ConnectionPool-Thread-25" com.DeliveryHandler -- message="INFO: First Time" timestamp="1553477992048"

both above events got merged and came as single event.

0 Karma

vijayakumarkb
Explorer

should i be using %Y-%m-%d %H:%M:%S:%3N

0 Karma

nickhills
Ultra Champion

Tbh, I’m not sure how , is handled vs . in the time format. Worth a try, I can’t see what else is wrong.
I presume the events are split by line in the actual source file?

If my comment helps, please give it a thumbs up!
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Use below props.conf, its almost same as provided by @nickhillscpl but timestamp config is fixed.

[yoursourcetype]
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%3N %z
TIME_PREFIX=^\[logLevel=\w+\]\s-\s
MAX_TIMESTAMP_LOOKAHEAD=29
LINE_BREAKER=(^)\[logLevel
0 Karma

yadavmalay
Observer

Logs are truncated at beginning

e WorkDay CW Data" transaction_start_epoch="1597257375.0102034" execution_id="87e92c54-dcca-11ea-8c01-0050568d9e34" browser="HeadlessChrome" browser_version="84.0.4147" os="Windows" os_version="10" ip="10.24.85.121" title="Horizon ACM - Custom Task: Synchronize  Data" app_name="XYZ

Below are the props.conf file I am using at universal forwarder side.

SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%3N %z
TIME_PREFIX=^\[logLevel=\w+\]\s-\s
MAX_TIMESTAMP_LOOKAHEAD=29
LINE_BREAKER=(^)\[logLevel

 

Any suggestion please.

0 Karma

vijayakumarkb
Explorer

Thanks Harsh for the help. let me test it

0 Karma

vijayakumarkb
Explorer

let me try with %Y-%m-%d %H:%M:%S,%N3

0 Karma

vijayakumarkb
Explorer

sorry %Y-%m-%d %H:%M:%S,%3N

0 Karma

vijayakumarkb
Explorer

I presume the events are split by line in the actual source file?

Yes events are split

0 Karma

nickhills
Ultra Champion

Yikes, you don't want events that are 5MB each!

If my comment helps, please give it a thumbs up!
0 Karma

vijayakumarkb
Explorer

Eg :

[logLevel=ERROR] - 2019-03-22 11:30:04,100 +0000 -- "ConsumerService-10-Thread-9" MasterService -- requestId="abc234" requestType="Register" message="Reporting - Success."
[logLevel=ERROR] - 2019-03-22 11:30:04,201 +0000 -- "ConsumerService-10-Thread-7" MasterService -- message="INFO SEND"

it is not coming as two separate events in Splunk always. getting trimmed of like as below

erService-10-Thread-29" MasterService -- requestId="abc234" requestType="Register" message="Reporting - Success."

0 Karma

nickhills
Ultra Champion

Can you post some example events, it sounds like you just need to configure your breaking settings correctly

If my comment helps, please give it a thumbs up!
0 Karma

vijayakumarkb
Explorer

[logLevel=ERROR] - 2019-03-22 11:30:04,100 +0000 -- "ConsumerService-10-Thread-9" MasterService -- requestId="abc234" requestType="Register" message="Reporting - Success."
[logLevel=ERROR] - 2019-03-22 11:30:04,201 +0000 -- "ConsumerService-10-Thread-7" MasterService -- message="INFO SEND"

it is not coming as two separate events in Splunk always. getting trimmed of like as below

erService-10-Thread-29" MasterService -- requestId="abc234" requestType="Register" message="Reporting - Success."

0 Karma

vijayakumarkb
Explorer

Any help for this

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...