Getting Data In

Handling Data with multiple formats

jhallman
Explorer

Has anyone worked with parsing multiple formats within a log

Example we logs like driver.log for our Datasynapse Grid processing
and at least 5 different distinct formats

mutil-line format

[LOG|DEBUG|2011 August 23, 08:25:27 (622)|MEMORY_DEBUG|ResponseCallbacks-1: DriverJobSpace$1|162.103.129.63 (wppsa01a0038.wellsfargo.com)]
In FuBaseWebProcJob::processTaskOuput(0) - heap size(50,577,408) free(8,822,904) % free(17.444357765427597)
[END]

2nd mutli-line
Bond has been loaded from Calypso
putting bond into cache cusip 3133XYJ97
SourceHit=22.0 CacheHit=5.0 HitRate=18.519
**** out of sync block
*********BondSettleDays =1 tradeSd=08/24/2011
::grName::gridlib_smiley2_prod_ro
Resetting DATASYNAPSE_RETRIES to 0
[2011-08-23 08:25:25.805] CARE Domain: MSRBTaskTimeoutMin=null
[2011-08-23 08:25:25.805] Executing grid job...

And at least 3 single line formats

08/23/11 08:25:27.627 INFO: [ServiceEvent] CompletedTask:TradeAnalyticsJob:3133XYJ97-8293306600710979712-0:Total:1

CARESERVICE END:CE0C1AE5-E762-4474-9541-E8724CFD8C86|45|S|3133XYJ97: TIME::8/23/11 11:59:00.674 PM EDT

CalypsoServiceGrid Response has been posted. 27.0#27.0

woodcock
Esteemed Legend

I assume the problem is that these variants are all inside of a single file. This blog does a good job of explaining how to handle that:

http://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...