Trying to Split one Event into Multiple Events

[jordankswan]

Hi All,

I am trying to split a Splunk event into multiple events. I just want each line to be an event, and it was my understanding that this is Splunk's default line breaking attitude as long as each line has a time stamp.

I am doing some TraceRoutes from various locations and having them output to a log file that I am sending to Splunk. I have been able to add a timestamp to each line and this made most of the lines be their own Splunk event, but the last 3 or 4 hops get bundled together into a single event.

Here is an example of the lines that Splunk is putting into a single event:

May 09 08:00:31 9.|-- fw.edge-sc9.elliemae.com 0.0% 30 78.4 78.5 78.2 81.7 0.6
May 09 08:00:31 10.|-- fw.edge-sc9.elliemae.com 0.0% 30 79.0 79.1 78.8 79.6 0.0
May 09 08:00:31 11.|-- be11147453.ea.elliemae.ne 0.0% 30 78.4 78.6 78.4 79.5 0.0

Note that each line for hops 1-8 have been split up into their own individual events.

Please let me know if I just need to make a simple edit to the props.conf or something else.

Thanks,

Jordan

[niketn · Accepted solution]

@jordankswan, Besides setting the SHOULD_LINEMERGE have you checked TIME_FORMAT and MAX_TIMESTAMP_LOOKAHEAD settings

[MTR]
SHOULD_LINEMERGE=false
TIME_FORMAT=%b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD=15
BREAK_ONLY_BEFORE_DATE=false

Since your Date field does not have Year I am expecting Splunk is not interpreting Date Time stamp correctly for some events and ends up merging them together.
Please try above props.conf. PS: You would need to test with a test index and also drop events which are not indexed correctly in your previous attempts. If this does not work, please post the sample events where Event Breaking/Timestamp recognition did not work as expected.

Also test us adding sample data file in the Data Preview mode to see upfront whether there are any issues with configurations files and sample data input. Refer to Documentation https://docs.splunk.com/Documentation/Splunk/latest/Data/Uploaddata (Please change documentation version as per your Splunk Enterprise version)

[xpac]

Just wondering - with SHOULD_LINEMERGE set to false, shouldn't it create single-line events and not merge at all, no matter if it finds a timestamp or not?

[jordankswan]

@niketnilay

Thanks for your suggestion. I tried to add those lines to my props.conf file, but it didn't fix my issue. The events were still being grouped exactly as they were before, in the body of my question above.

However, thank you so much for explaining why it was most likely doing this, as you were correct.

I have control over how these log files are timestamping each line, so I just had to add the year in there and that fixed the issue.

Thank you so much!

Jordan

[xpac]

Hey,

SHOULD_LINEMERGE = [true|false]
* When set to true, Splunk combines several lines of data into a single
  multi-line event, based on the following configuration attributes.
* Defaults to true.

If you set that to false for your sourcetype, every line will be one event.

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

[jordankswan]

Actually, false alarm, it worked for two pulls of the logs but it is back to doing the same thing even though I have made changes to the props.conf file and restarted our Splunk server.

The sourcetype for these logs is [MTR]

and here is the entry in my props.conf file:
[MTR]
SHOULD_LINEMERGE = false

Thanks.

[xpac]

Go to the CLI and try a splunk bool props list MTR and see what value is set there, please.

[MuS]

Nice hint using btool, but here is a BUT : you must be aware that btool does not necessarily shows the config Splunk is using. Here is the quote from the docs http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Usebtooltotroubleshootconfigurati...

Btool displays merged on-disk configurations. That is, btool shows you the merged settings in the       .conf files. It does not necessarily show you what Splunk software is currently using. 

cheers, MuS

[jordankswan]

Here are the results of that command:

[MTR]
ADD_EXTRA_TIME_FIELDS = True
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG = /etc/datetime.xml
DEPTH_LIMIT = 1000
HEADER_MODE =
LEARN_MODEL = true
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MATCH_LIMIT = 100000
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = false
TRANSFORMS =
TRUNCATE = 10000
detect_trailing_nulls = false
maxDist = 100
priority =
sourcetype =

[xpac]

You're absolutely right, you could use splunk show config propsand would than have to look for the MTR stanza, that would be the actual running config.
When SHOULD_LINEMERGE is false and you didn't change the default linebreaker, it should really not create multi-line events...

[jordankswan]

Thats what I was thinking... I thought that it shouldn't be creating multi-line events....

Here are the results from the "splunk show config props" cmd.

[MTR]
ADD_EXTRA_TIME_FIELDS=True
ANNOTATE_PUNCT=True
AUTO_KV_JSON=true
BREAK_ONLY_BEFORE_DATE=True
CHARSET=UTF-8
DATETIME_CONFIG=/etc/datetime.xml
DEPTH_LIMIT=1000
LEARN_MODEL=true
LEARN_SOURCETYPE=true
LINE_BREAKER_LOOKBEHIND=100
MATCH_LIMIT=100000
MAX_DAYS_AGO=2000
MAX_DAYS_HENCE=2
MAX_DIFF_SECS_AGO=3600
MAX_DIFF_SECS_HENCE=604800
MAX_EVENTS=256
MAX_TIMESTAMP_LOOKAHEAD=128
SEGMENTATION=indexing
SEGMENTATION-all=full
SEGMENTATION-inner=inner
SEGMENTATION-outer=outer
SEGMENTATION-raw=none
SEGMENTATION-standard=standard
SHOULD_LINEMERGE=false
TRUNCATE=10000
detect_trailing_nulls=false
maxDist=100

[jordankswan]

That did that trick.

Thank you so much, didn't realize it was such an easy fix.