Hi all,
I don’t know exactly how long this has been going on but I noticed today that the following error is being spammed into the /opt/splunkforwarder/var/log/splunk/splunkd.log file on our system.
04-22-2019 17:36:10.474 -0700 ERROR JsonLineBreaker - JSON StreamId:14919777892573414995 had parsing error:Unexpected character: '4' - data_source="/opt/splunkforwarder/var/log/splunk/splunkd.log", data_host="smn-sn-util01", data_sourcetype="splunkd"
04-22-2019 17:36:10.474 -0700 ERROR JsonLineBreaker - JSON StreamId:14919777892573414995 had parsing error:Unexpected character: '4' - data_source="/opt/splunkforwarder/var/log/splunk/splunkd.log", data_host="smn-sn-util01", data_sourcetype="splunkd"
There's also this error which occasionally pops up in the log but not nearly as frequently as the one above:
04-22-2019 17:47:05.009 -0700 ERROR JsonLineBreaker - JSON StreamId:8970828008188520838 had parsing error:Unexpected character while looking for value: 'A' - data_source="/var/log/messages", data_host="smn-sn-util01", data_sourcetype="syslog"
04-22-2019 17:47:05.009 -0700 ERROR JsonLineBreaker - JSON StreamId:8970828008188520838 had parsing error:Unexpected character while looking for value: 'A' - data_source="/var/log/messages", data_host="smn-sn-util01", data_sourcetype="syslog"
I tried stopping splunk, removing all the splunkd.log* files, and then restarting splunk but the error continues to show up in the logs. Any ideas as to what may be causing this?
You removed splunkd.log file from host smn-sn-util01
? Also, you can run btool command on that host for splunkd sourcetype and provide output here?
/opt/splunkforwarder/bin/splunk btool props list splunkd --debug | grep -v system/default
Yup, I removed splunkd.log from smn-sn-util01 and the issue persisted. Here's the output of the command:
/opt/splunkforwarder/etc/apps/search/default/props.conf [splunkd]
/opt/splunkforwarder/etc/system/local/props.conf AUTO_KV_JSON = false
/opt/splunkforwarder/etc/apps/search/default/props.conf EXTRACT-fields = (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+) - (?P<message>.+)
/opt/splunkforwarder/etc/system/local/props.conf INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf KV_MODE = none
/opt/splunkforwarder/etc/system/local/props.conf TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf KV_MODE = none
/opt/splunkforwarder/etc/system/local/props.conf TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf TIME_FORMAT = %s
/opt/splunkforwarder/etc/system/local/props.conf TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf KV_MODE = none
/opt/splunkforwarder/etc/system/local/props.conf TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf TIME_FORMAT = %s
/opt/splunkforwarder/etc/system/local/props.conf TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf KV_MODE = none
/opt/splunkforwarder/etc/system/local/props.conf TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf TIME_FORMAT = %s
/opt/splunkforwarder/etc/system/local/props.conf TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf KV_MODE = none
/opt/splunkforwarder/etc/system/local/props.conf TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf KV_MODE = none
/opt/splunkforwarder/etc/system/local/props.conf TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf TIME_FORMAT = %s
/opt/splunkforwarder/etc/system/local/props.conf TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf KV_MODE = none
/opt/splunkforwarder/etc/system/local/props.conf TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf TIME_FORMAT = %s
/opt/splunkforwarder/etc/system/local/props.conf TZ = UTC