Splunk IT Service Intelligence

Splunk App for Infrastructure, CentOS 7, JSON exception caught while processing collectd event: Unexpected character: 'm'

asucrews
Path Finder

Both servers is CentOS 7
One with Splunk Enterprise 7.2.5
Splunk App for Infrastructure 1.2.3
Splunk Add-on for Infrastructure 1.2.3

one with Splunk Universal Forwarder 7.2.5

Error message: 03-20-2019 13:52:05.257 -0700 ERROR MetricsProcessor - JSON exception caught while processing collectd event: Unexpected character: 'm'
03-20-2019 13:52:05.257 -0700 ERROR MetricsProcessor - Failed to parse metrics input, most likely due to incorrect protocol JSON exception caught while processing collectd event: Unexpected character: 'm'
"

Hello, I am trying out Splunk App for Infrastructure and at this time all i am getting is the above error message. And i not sure if something changed in collectd or app. I have read thought the docs online but can't seem to find any thing like this. Has anyone ran in to this issue before.

0 Karma

asucrews
Path Finder

I have narrowed it down the issue to write_splunk plugin, when i switch to write_http plugin i start getting data. Not sure if this effect the dashboards but it is now working.

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

I have seen the same issue before. The issue was with the hec_token not being set properly with correct sourcetype OR Add on for Infra not installed. Are you sending collectd data directly to SAI or using any forwarder in between?

For write_http to work for you, it needs collectd_http as the sourcetype. Did you change that after switching to write_http?

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

Sourcetype should be em_metrics not Automatic.

0 Karma

asucrews
Path Finder

disabled ssl, and change sourcetype to em_metrics. still getting "03-21-2019 09:13:51.651 -0700 ERROR MetricsProcessor - JSON exception caught while processing collectd event: Unexpected character: 'm'"
"03-21-2019 09:13:51.651 -0700 ERROR MetricsProcessor - Failed to parse metrics input, most likely due to incorrect protocol JSON exception caught while processing collectd event: Unexpected character: 'm'"

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

Could you please check the hec token that you created?

Make sure sourcetype and index is "em_metrics" for the token.

https://docs.splunk.com/Documentation/InfraApp/1.2.3/Install/Install

0 Karma

asucrews
Path Finder

Deleted are readded HED per https://docs.splunk.com/Documentation/InfraApp/1.2.3/Install/Install still same error

0 Karma

asucrews
Path Finder

collectd 5.8.1, http://collectd.org/
by Florian octo Forster
for contributions see `AUTHORS'

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

Did you run the script on your terminal from "Add Data" page on Splunk App for Infra ?

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

Did you restart Splunk after installing "Splunk Add on for Infra" ?

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

ALso, rerun the script with right hec_token if you have deleted and created a new token

0 Karma

asucrews
Path Finder

deleted and recreated HEC . still same error.

Review
Input Type Token
Name SPI_HEC
Source name override N/A
Description N/A
Enable indexer acknowledgements No
Output Group N/A
Allowed indexes em_metrics
Default index em_metrics
Source Type Automatic
App Context splunk_app_infrastructure

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...