Hi, I've reviewed almost all the question about event line breaking but still have some inconsistency with data ingesting to my Splunk Enterprise. Is there any sort of debugging/logging system for data input and the method which Splunk use to handle multiline events. My data looks like this:
{"timestamp_ns":1497623896051426216,"timestamp":1497623896,"measurements":{"response_send_time_us":147,"walltime_us":1003,"xxxxx_walltime_us":493,"xxxxx_walltime_us":510,"xxxxx_time_us":159,"xxxxx_time_us":120,"xxxxx_time_us":82},"application":"xxxxx","type":"xxxxx_query_request","metadata":{"request_type":2,"xx_id":1,"request_timestamp_ns":1497623896050422653,"request_id":"1234567890123456789"}}
{"timestamp_ns":1497623896051426216,"timestamp":1497623896,"measurements":{"response_send_time_us":147,"walltime_us":1003,"xxxxx_walltime_us":493,"xxxxx_walltime_us":510,"xxxxx_time_us":159,"xxxxx_time_us":120,"xxxxx_time_us":82},"application":"xxxxx","type":"xxxxx_query_request","metadata":{"request_type":2,"xx_id":1,"request_timestamp_ns":1497623896050422653,"request_id":"1234567890123456789"}}
{"timestamp_ns":1497623896051426216,"timestamp":1497623896,"measurements":{"response_send_time_us":147,"walltime_us":1003,"xxxxx_walltime_us":493,"xxxxx_walltime_us":510,"xxxxx_time_us":159,"xxxxx_time_us":120,"xxxxx_time_us":82},"application":"xxxxx","type":"xxxxx_query_request","metadata":{"request_type":2,"xx_id":1,"request_timestamp_ns":1497623896050422653,"request_id":"1234567890123456789"}}
{"timestamp_ns":1497623896051426216,"timestamp":1497623896,"measurements":{"response_send_time_us":147,"walltime_us":1003,"xxxxx_walltime_us":493,"xxxxx_walltime_us":510,"xxxxx_time_us":159,"xxxxx_time_us":120,"xxxxx_time_us":82},"application":"xxxxx","type":"xxxxx_query_request","metadata":{"request_type":2,"xx_id":1,"request_timestamp_ns":1497623896050422653,"request_id":"1234567890123456789"}}
Here's what I've tried in props.conf:[SOURCETYPE_NAME]
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+){\"timestam_ns\"
TIME_FORMAT = %s%9N
TIME_PREFIX = "timestamp_ns":
ForSHOULD_LINEMERGE = false
, I've tried otherLINE_BREAKER
as well, like^\{
,([\r\n]+)\{
, etc but no luck.
[SOURCETYPE_NAME]
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ^\{
TIME_FORMAT = %s%9N
TIME_PREFIX = "timestamp_ns":
Just for your information, I've tried manual Add Data feature and both configs works fine there. And I'm testing these stuff on a Splunk Developer Personal License before applying the changes against the actual Enterprise version.
I would give this a try. Also ensure that you kept this config in right place (Indexer/heavy forwarder whichever comes first in flow)
[SOURCETYPE_NAME]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\s*\{\s*\"timestam_ns\")
TIME_FORMAT = %s%9N
TIME_PREFIX = ^\s*\{\s*\"timestam_ns\"
MAX_TIMESTAMP_LOOKAHEAD = 20
Hi msichani,
try the following stanza
[SOURCETYPE_NAME]
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = \{\"timestamp\_ns\"
TIME_FORMAT = %s%9N
TIME_PREFIX = "timestamp_ns":
The problem was with HF in between, your stanaza should definitely work as well but with proper routing. Thanks anyway.
I would give this a try. Also ensure that you kept this config in right place (Indexer/heavy forwarder whichever comes first in flow)
[SOURCETYPE_NAME]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\s*\{\s*\"timestam_ns\")
TIME_FORMAT = %s%9N
TIME_PREFIX = ^\s*\{\s*\"timestam_ns\"
MAX_TIMESTAMP_LOOKAHEAD = 20
I think the trick was the right place, it was going through heavy forwarder, Added _TCP_ROUTING
and it looks fine now. Thanks for the hint @somesoni2