Getting Data In

How to fix line breaking issue Unix timestamp

samadmemon
Explorer

Hi All,

I have the logs in below format which is stored in an S3 bucket :

1567295878959445,hostname,ip,id,session,operation,db,query

The first field I believe is the Unix timestamp. When I am integrating those logs with Splunk Addon for AWS the line breaking is not happening as per the timestamp.
Below is the sample log that I am receiving in Splunk. The log is not breaking based on the timestamp as you can see below:

1567295878959445,hostname,ip,id,session,operation,db,query,1567295878959550,hostname,ip,id,session,operation,db,query'

Could anyone advise the configuration in props.conf to break these logs as per the timestamp? Ideally, log should look like below in Splunk :

1567295878959445,hostname,ip,id,session,operation,db,query -log1

1567295878959550,hostname,ip,id,session,operation,db,query' -log2

Regards,
Samad

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Give this a try (props.conf on your heavy forwarder/indexer, first Splunk Enterprise instance that comes in your data flow)

[YourSourcetypeHere]
LINE_BREAKER = ([\r\n]+)(?=\d+,)
SHOULD_LINEMERGE = false
TIME_PREFIX = ^
TIME_FORMAT = %s.%6N

Would require Splunk restart to take effect.

samadmemon
Explorer

Thank you for the answer.

However, the above setting is not working.

0 Karma

rupesh26
Path Finder

Try this
TIME_FORMAT = %s%3N

0 Karma

samadmemon
Explorer

It is now taking the timestamp without any errors, however the events are not breaking based on the timestamp. I used below setting :

LINE_BREAKER = (\d{10,16}+)

It is breaking the events but it is removing timestamp from the logs.

Regards,
Samad

0 Karma

rupesh26
Path Finder

doesn't this LINE_BREAKER work for you ?

LINE_BREAKER = ([\r\n]+)(?=\d+,)

0 Karma

samadmemon
Explorer

Hi Rupesh,

No, this line breaker doesn't work.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

So, all the logs are in single lines thus the my line breaker didn't work. Logs like this with no separator between the events are tough to break. Do you have control to change the logging format, either changing it to single line OR including some static/unique characters before the event?

0 Karma

samadmemon
Explorer

I don't have control to change the logging format. However, I will check with admin to see if there is any possibility to change the logging format.

Thanks for your help.

0 Karma

maciep
Champion

what about something like this for your line breaker....assuming you don't need the comma

(,\s*)(?=\d{10})
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...