Hello Splunk Community,
I have an issue with JSON parsing in Splunk and hope you can help me with that.
Situation:
[monitor:///here_is_the_correct_path]
disabled = false
host_segment = 3
index = buttercup
sourcetype = buttercup:server
host =
[buttercup:server]
TZ = UTC
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
description = buttercup Server Logs
pulldown_type = true
TRANSFORMS-afilter = setnull, setparsing-audit, setparsing-auth
TRANSFORMS-changesourcetype = change-buttercup-server-audit, change-buttercup-server-auth
[buttercup:server:audit]
TZ = UTC
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
description = buttercup Server Auditlog
pulldown_type = true
SEDCMD-strip_prefix = s/^[^{]+//g
[buttercup:server:auth]
TZ = UTC
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
description = buttercup Server Authenticationlog
pulldown_type = true
[change-buttercup-server-audit]
REGEX = buttercup_audit\:
FORMAT = sourcetype::buttercup:server:audit
DEST_KEY = MetaData:Sourcetype
[change-buttercup-server-auth]
REGEX = buttercup_auth\:
FORMAT = sourcetype::buttercup:server:auth
DEST_KEY = MetaData:Sourcetype
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing-audit]
REGEX = buttercup_audit\:
DEST_KEY = queue
FORMAT = indexQueue
[setparsing-auth]
REGEX = buttercup_auth\:
DEST_KEY = queue
FORMAT = indexQueue
Description:
Problem:
I think the problem has something to do with the SEDCMD and when it will be handled. Do you have any idea, how to fix that? I was thinking about doing the SEDCMD part within an additional transforms instead but I don´t know how. I have also experimented with adding INDEXED_EXTRACTIONS=JSON and KV_MODE=none (and vise versa) to the sourcetype, but no success.
Hello all,
we' ve solved the problem:
Props.conf - sourcetype buttercup:server was adjusted
[buttercup:server]
TZ = UTC
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
description = buttercup Server Logs
pulldown_type = true
TRANSFORMS-afilter = setnull, setparsing-audit, setparsing-auth
TRANSFORMS-changesourcetype = change-buttercup-server-audit, change-buttercup-server-auth
TRANSFORMS-strip-front-json = strip-front-json
The last line was added and this replaced also the SEDCMD command in the buttercup:server:audit sourcetype.
transforms.conf - added the following
[strip-front-json]
REGEX = ((?<=buttercup_audit: )(.*))
FORMAT = $2
DEST_KEY = _raw
The main problem was, that after rewriting the sourcetype and filtering the events, the parsing does not have any effect anymore because it was already in the indexQueue.
SEDCMD whould have also worked in the sourcetype buttercup:server we think.
There is maybe a better/more efficient way of orderng the transforms, but it works for now.
Many Regards
Michael