Getting Data In

How to prevent splunk from merging few JSON strings into single event?

spellanser
Explorer

Example raw data:

{"field1": "value1", "field2": "value2", ..., "string": "1" } 
{"field1": "value1", "field2": "value2", ... ,"string":"2"} 
{"field1": "value1", "field2": "value2", ..., "string":"3" }
{"field1": "value1", "field2": "value2", ..., "string":"4" }

Splunk merge few of raw data string into single event, as result you got 2 events.
Event 1:

{"field1": "value1", "field2": "value2", ..., "string": "1" } 
{"field1": "value1", "field2": "value2", ... ,"string":"2"} 
{"field1": "value1", "field2": "value2", ..., "string":"3" }

Event 2:

{ [-]
    field1: value1
    field2: value2
     ...
    string: 4
}

So, 80% of events looks like event 1 in example. But some events caught by single row and parsed as JSON type.

Using cluster of Splunk Enterprise and splunkforwarder for data delivery, version 6.5.5.

I have tried to setup props.conf on splunkforwarder (app which work with JSON log files), tried to use different LINE_BREAKER:

1. (\})
2. \}
3. "(^)\{" 

Current props.conf:

[json-logs]
SHOULD_LINEMERGE = false
KV_MODE = json
LINE_BREAKER = (\})
TIME_PREFIX = \"time\": \"

Have same problem not only with JSON format logs, looks like props.conf line breaking options not work at all.
What I am doing wrong?

0 Karma
1 Solution

s2_splunk
Splunk Employee
Splunk Employee

props.conf settings have to be present where the event parsing occurs. The forwarder (if it is a universal forwarder) does NOT parse events. If you have a UF sending your data to your indexer(s), put your props.conf for the sourcetype on the indexer(s).

While a bit dated, this Wiki article still contains accurate information on what settings apply to which phase of the event processing pipeline.

You can also try to use BREAK_ONLY_BEFORE=^\{ instead of LINE_BREAKER if you are certain that all your events start with a '{' at the beginning of a line.

View solution in original post

syunwei
Engager

Hi spellanser,

I had the same problem with that merged JSON logs.
I've tried various props.conf setting that were not working until I change the TIME_PREFIX, TIME_FORMAT and MAX_TIMESTAMP_LOOKAHEAD settings.
My guess is when Splunk try to parsing logs and couldn't find event timestamp so then the logs were not splitted.

I can see from Splunk document:
TIME_PREFIX:
* If the TIME_PREFIX cannot be found in the event text, timestamp extraction
will not occur.
* Defaults to empty.

Here is my JSON logs look like:

{"asctime": "2018-06-22T09:13Z+0000", "exception": "xxxx", "function_name": "xxxx"}
{"asctime": "2018-06-22T09:15Z+0000", "exc_duration": 100, "exc_memory": "70 MB"}

Props.conf:

[my_sourcetype]
INDEXED_EXTRACTIONS = json
KV_MODE=none
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=([\r\n]+)
TIME_PREFIX=asctime:\s
MAX_TIMESTAMP_LOOKAHEAD=25
TIME_FORMAT=%Y-%m-%dT%H:%M:%SZ%z

Hope this may help in your situation.
Cheers

0 Karma

spellanser
Explorer

Thank you, will try it!

0 Karma

burwell
SplunkTrust
SplunkTrust

Hello Spellanser: Did you have any luck?

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

props.conf settings have to be present where the event parsing occurs. The forwarder (if it is a universal forwarder) does NOT parse events. If you have a UF sending your data to your indexer(s), put your props.conf for the sourcetype on the indexer(s).

While a bit dated, this Wiki article still contains accurate information on what settings apply to which phase of the event processing pipeline.

You can also try to use BREAK_ONLY_BEFORE=^\{ instead of LINE_BREAKER if you are certain that all your events start with a '{' at the beginning of a line.

spellanser
Explorer

Thank you for answer. I was confused with this in wiki article, which you mentioned:

Since splunk 6, some source can be parsed for structured data (like headers, or json) and be populated at the forwarder level. see http://docs.splunk.com/Documentation/Splunk/6.1.2/Data/Extractfieldsfromfileheadersatindextime#Forwa... Those setting have to be on the forwarders (and indexers if they monitor files)

Looks like it's not work at all.

Also i have question, what is the right way to distribute props.conf to indexers on cluster of Splunk Enterprise? Using master and master-apps or creating new custom application?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...