Getting Data In

How to break multi-line event logfile so event starts when date found at beginning of a line?

a212830
Champion

Hi,

I have a multi-line event logfile that I'm having issues with. I want to say that an event starts when it finds date at the beginning of a line. The output looks like this (the number of lines between dates can vary widely - this is just a sample.

09 Jan 2014 02:02:13,477 [TaskEngineWorker-pool-1] [::] INFO tasks.AnalyticsCachePrimerETL - AnalyticsCachePrimerETL: Task started
09 Jan 2014 02:02:15,436 [TaskEngineWorker-pool-1] [::] INFO tasks.AnalyticsCachePrimerETL - AnalyticsCachePrimerETL: Task completed in 1958 milliseconds
09 Jan 2014 02:02:42,087 [TaskEngineWorker-pool-1] [::] INFO impl.VantageAsyncRestCall -
ParentResourceId=;
ResourceDisplayName=;
ResourceId=2020-78323;
ResourceType=USERCONTAINER;
ActionType=EDIT;
ApplicationId=1000000;
EventTime=1389250824323;
EventType=DISCUSSION;
Modality=COLLAB;
FirstName=;
LastName=;
Managed=true;
MimeType=html;
09 Jan 2014 02:25:40,424 [TaskEngineWorker-pool-1] [::] ERROR emailwhitelist.WhitelistEmailManagerImpl - UAT EMAIL WHITELIST PLUGIN INSTALLED. ONLY ALLOWING EMAIL TO A RESTRICTED SET OF USERS!!!!
09 Jan 2014 02:25:40,424 [TaskEngineWorker-pool-1] [::] ERROR emailwhitelist.WhitelistEmailManagerImpl - UAT EMAIL WHITELIST PLUGIN INSTALLED. ONLY ALLOWING EMAIL TO A RESTRICTED SET OF USERS!!!!

Can someone help me?

Tags (1)
0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

It's hard to tell where the actual line breaks are in your file... (whether we are seeing real ones or a wrap here)
But basically you want something like this:

BREAK_ONLY_BEFORE=\d{2}\s+[JFMASOND][aepuco][nbrylgptvc]\s+\d{4}\s\d{2}:\d{2}:\d{2},\d{3}
MAX_TIMESTAMP_LOOKAHEAD=30
SHOULD_LINEMERGE=true

I have literally "spelled out" the structure of the timestamp. Being so specific is probably overkill...
However, to line break on a multiline event file. You specify something that will allow Splunk to see where to break. Tell it where the timestamp is and then tell it to merge the lines in between the "markers".

Also the way I've done it... it doesn't care about or use any actual carriage returns. However if you have them, and you can count on them showing up in a consistent place, you can use them too...
If you bring a sample into the Previewer, you will see how these things are applied.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

lguinn2
Legend

What is in props.conf right now? What sourcetype is assigned to this data?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...