Getting Data In

props.conf help

a212830
Champion

Hi,

I have a multi-line feed that I'm having problems with - hoping someone can help me. Here's my props.conf:

MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TIME_PREFIX = ^
TRUNCATE = 999999
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\d{4}\s\w{2}\s\d{2}

And, here's some sample data. The break is not occurring on the lines properly. Pretty sure my LINE_BREAKER is wrong. On a side note, if I just let splunk do the work, everything is wonderful, but this is not how we want to eng our systems. Is there any way to tell what splunk is applying for timestamps when you don't specify a props.conf?

2014-07-23 02:44:46,913 INFO [com.mpi.saml.plugin.SamlTokenRepositoryCleanupTask] Deleting the expired tokens from SAML repository
2014-07-23 02:46:21,620 INFO [com.media3k.escene.util.background.SsoNonceCleaner] DEFAULT.SsoNonceCleaner executing at Wed Jul 23 02:46:21 EDT 2014
2014-07-23 02:59:46,914 INFO [com.mpi.saml.plugin.SamlTokenRepositoryCleanupTask] Deleting the expired tokens from SAML repository
2014-07-23 03:01:21,693 INFO [com.media3k.escene.util.background.SsoNonceCleaner] DEFAULT.SsoNonceCleaner executing at Wed Jul 23 03:01:21 EDT 2014
2014-07-23 03:06:21,621 INFO [com.media3k.escene.util.background.SsoNonceCleaner] DEFAULT.SsoNonceCleaner executing at Wed Jul 23 03:06:21 EDT 2014
2014-07-23 03:14:15,379 ERROR [com.media3k.escene.util.background.TempFileCleaner] Unable to find ingesting Programs to activate.
java.lang.NullPointerException: null
at com.qumu.prg.builder.ProgramAssociationService.findProgramContentAssociation(ProgramAssociationService.java:168) ~[mpi-bcast.jar:na]
at com.media3k.escene.util.background.TempFileCleaner.processImage(TempFileCleaner.java:115) [adminportal.jar:na]
at com.media3k.escene.util.background.TempFileCleaner.access$200(TempFileCleaner.java:39) [adminportal.jar:na]
at com.media3k.escene.util.background.TempFileCleaner$1.call(TempFileCleaner.java:70) [adminportal.jar:na]
at com.media3k.escene.util.background.TempFileCleaner$1.call(TempFileCleaner.java:57) [adminportal.jar:na]
at com.mpi.framework.threadcontext.ThreadContexts$ContextCallableImpl.call(ThreadContexts.java:141) [foundation.jar:na]
at com.media3k.escene.util.background.TempFileCleaner.finalizeEgcPrograms(TempFileCleaner.java:57) [adminportal.jar:na]
at com.media3k.escene.util.background.TempFileCleaner.executeInternal(TempFileCleaner.java:51) [adminportal.jar:na]
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:113) [spring-context-support.jar:3.2.1.RELEASE]
at org.quartz.core.JobRunShell.run(JobRunShell.java:216) [quartz.jar:na]
at com.media3k.escene.util.background.TaskExecutionServiceThreadPool$QuartzTask.call(TaskExecutionServiceThreadPool.java:93) [mpi-lodgepole.jar:na]
at com.media3k.escene.util.background.TaskExecutionServiceThreadPool$QuartzTask.call(TaskExecutionServiceThreadPool.java:82) [mpi-lodgepole.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_65]
at com.mpi.framework.task.impl.juc.JUCThreadPool$InternalFutureTask.run(JUCThreadPool.java:328) [foundation.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
2014-07-23 03:14:15,379 INFO [com.media3k.escene.util.background.TempFileCleaner] CentralConfig.filerTempDir not set or does not exist; using D:\Temp for file upload directory.
2014-07-23 03:14:15,380 INFO [com.media3k.escene.util.background.TempFileCleaner] Cleaning up the temp folder: D:\Temp
2014-07-23 03:14:46,915 INFO [com.mpi.saml.plugin.SamlTokenRepositoryCleanupTask] Deleting the expired tokens from SAML repository
2014-07-23 03:21:21,634 INFO [com.media3k.escene.util.background.SsoNonceCleaner] DEFAULT.SsoNonceCleaner executing at Wed Jul 23 03:21:21 EDT 2014

Tags (2)
0 Karma

linu1988
Champion

Hello,
The below would actually be appropriate in your case.

NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TRUNCATE = 0
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = true

OR

NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TRUNCATE = 0
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE =\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}

Thanks,
L

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Your second LINE_BREAKER string should do it. If not, try '([\r\n]+)(\d+)-(\d+)-(\d+)'

---
If this reply helps you, Karma would be appreciated.
0 Karma

a212830
Champion

Oh, maybe...?

MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TIME_PREFIX = ^
TRUNCATE = 999999
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...