Splunk Search

How do I split one event on my Splunk instance into multiple events?

swinte12
New Member

I have several indexes in my Splunk Instance. One of these instances is merging some of my log events into a single event incorrectly. I need help splitting them up. At the bottom of this question, I have posted an example. What I need help with is... what is the easiest way to split these events so that each event is separate? I believe that the best way is to specify the attributes in the props.conf file to change the line breaking and merging behavior so that all lines are merged ("SHOULD_LINEMERGE = true") and then set the attribute to Break the lines at the beginning when the date is given ("BREAK_ONLY_BEFORE = ^(?P[^ ]+)") - not sure my regex is correct.

  1. is this the correct way to resolve my problem?
  2. do I change this in the Splunk Enterprise Local props.conf file, or do I make this change on the Forwarder for this particular index?
  3. how do I ensure it does not change the behavior of other indexes or sourcetypes?
  4. is this regex correct for what I am trying to do? I am trying to break whenever the timestamp exists at the beginning like 18:48:56. It is always in the same format ##:##:##. sometimes there are stacktrace or multiline logs in one event, that is why I need them to be merged and seperated differently.

these lines are recognized as separate events

    18:48:55 Completed Teststep: successful login using email: {0} and password: {1} with arguments: [swinters@gci.com, Ch0c0late$tarfish]
    18:48:56 Invoking Teststep: get {url} with arguments: []
    18:48:56 Invoking Teststep: get {url} with arguments: []
    18:48:56 Invoking Teststep: get {url} with arguments: 
**these lines are not recognized as separate events but should break after the timestamp
18:45:59 Failure Teststep: First successful login after deploy using email: {0} and password: {1} with arguments: [swinters@gci.com, Ch0c0late$tarfish]
com.qmetry.qaf.automation.step.JavaStepReporter$JPThrowable: org.openqa.selenium.TimeoutException: Expected condition failed: Wait time out for User Menu to be present (tried for 60 second(s) with 1000 MILLISECONDS interval)
Build info: version: '3.5.3', revision: 'a88d25fe6b', time: '2017-08-29T12:42:44.417Z'
System info: host: '52dc70ddb77f', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-123.8.1.el7.x86_64', java.version: '1.8.0_92'
Driver info: driver.version: unknown
    at com.qmetry.qaf.automation.step.JavaStepReporter$MockJavaStep.doExecute(JavaStepReporter.java:144)
    at com.qmetry.qaf.automation.step.BaseTestStep.execute(BaseTestStep.java:149)
    at com.qmetry.qaf.automation.step.JavaStepReporter.javaTestStep(JavaStepReporter.java:88)
    at hubTesting.steps.globalSteps.LoginSteps.firstLogin(LoginSteps.java:15)
    at hubTesting.test.globalTests.RefreshHubUser.refreshUserMgmtUser(RefreshHubUser.java:79)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:571)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:707)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:979)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.openqa.selenium.TimeoutException: Expected condition failed: Wait time out for User Menu to be present (tried for 60 second(s) with 1000 MILLISECONDS interval)
Build info: version: '3.5.3', revision: 'a88d25fe6b', time: '2017-08-29T12:42:44.417Z'
System info: host: '52dc70ddb77f', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-123.8.1.el7.x86_64', java.version: '1.8.0_92'
Driver info: driver.version: unknown
    at org.openqa.selenium.support.ui.FluentWait.timeoutException(FluentWait.java:296)
    at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:265)
    at com.qmetry.qaf.automation.ui.webdriver.QAFExtendedWebElement.waitForPresent(QAFExtendedWebElement.java:499)
    at com.qmetry.qaf.automation.step.CommonStep.waitForPresent(CommonStep.java:805)
    at hubTesting.steps.globalSteps.LoginSteps.firstLogin_aroundBody0(LoginSteps.java:23)
    at hubTesting.steps.globalSteps.LoginSteps$AjcClosure1.run(LoginSteps.java:1)
    at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
    at com.qmetry.qaf.automation.step.JavaStepReporter$MockJavaStep.doExecute(JavaStepReporter.java:138)
    ... 17 more
18:45:59 Completed Teststep: First successful login after deploy using email: {0} and password: {1} with arguments: [swinters@gci.com, Ch0c0late$tarfish]
18:46:00 Invoking Teststep: get {url} with arguments: []
18:46:00 Invoking Teststep: get {url} with arguments: []
18:46:00 Failure Teststep: First successful login after deploy using email: {0} and password: {1} with arguments: [swinters@gci.com, Ch0c0late$tarfish]
com.qmetry.qaf.automation.step.JavaStepReporter$JPThrowable: org.openqa.selenium.TimeoutException: Expected condition failed: Wait time out for User Menu to be present (tried for 60 second(s) with 1000 MILLISECONDS interval)
0 Karma
1 Solution

micahkemp
Champion

You don't need SHOULD_LINEMERGE, just LINE_BREAKER. This should split lines into new events only when they start with a timestamp:

props.conf:

[<sourcetype>]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\d{2}:\d{2}:\d{2}

Oh, and my regex assumes that the leading space on your log lines isn't actually present in the actual data, but an artifact of your copy/paste into answers.

View solution in original post

0 Karma

skoelpin
SplunkTrust
SplunkTrust

You should apply base configs in your props.conf and rerstart the Splunk service

Try this

[sourcetype]
TIME_PREFIX = ^
TIME_FORMAT = %H:%M:%S
SHOULD_LINE_MERGE = false
LINE_BREAKER = ([\r\n]+)(\s+\d+:\d+:\d+)
MAX_TIMESTAMP_LOOKAHEAD = 15
TRUNCATE = 1000

Make sure to replace sourcetype with your sourcetype. Also, is there a space in front of the hour? If so then the above configs will work

0 Karma

micahkemp
Champion

You don't need SHOULD_LINEMERGE, just LINE_BREAKER. This should split lines into new events only when they start with a timestamp:

props.conf:

[<sourcetype>]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\d{2}:\d{2}:\d{2}

Oh, and my regex assumes that the leading space on your log lines isn't actually present in the actual data, but an artifact of your copy/paste into answers.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...