Getting Data In

Why am I getting error "Unexpected character while looking for value: '<'" using Indexer Discovery in a master node of an indexer cluster?

asimagu
Builder

hi guys

I am just having a go with the new feature of Indexer Discovery at the master node of my 6.3 cluster.

I configured the following things:

In the master, server.conf I added:

[indexer_discovery]

In my Heavy Forwarder, I created the following outputs.conf under $SPLUNK_HOME/etc/apps/SplunkForwarder/local/

[indexer_discovery:master1]
master_uri = https://<my_ip>:8089

[tcpout:group1]
autoLBFrequency = 30
forceTimebasedAutoLB = true
indexerDiscovery = master1
useACK=true

[tcpout]
defaultGroup = group1
forwardedindex.filter.disable = true

I was trying to forward my _internal data, however, I am getting the following errors on splunkd.log

09-30-2015 16:45:50.639 +0100 ERROR HttpClientRequest - Caught exception while parsing HTTP reply: Unexpected character while looking for value: '<'
09-30-2015 16:45:50.639 +0100 ERROR IndexerDiscoveryHeartbeatThread - failed heartbeat for group=group1 uri=https://<my_ip>:8089/services/indexer_discovery http_response=Unauthorized

So, it's quite clear that there is a problem when the forwarder needs to contact the Indexer Discovery feature on the master.

When I try to browse https://<my_ip>:8089/services/indexer_discovery I get the following screen
alt text

Can you help me? What am I missing??

halayli_splunk
Splunk Employee
Splunk Employee

You need to set pass4SymmKey in the forwarder's [indexer_discovery] to match either [general] or [indexer_discovery]'s pass4SymmKey of CM.

On the other hand, your curl request is not valid. It should look something similar to this (It's a post and needs authentication):

curl -k -u admin:changeme -d "site=default" -d "guid=xxxx" https://localhost:8090/services/indexer_discovery

asimagu
Builder

I did not set any [indexer_discovery]'s pass4SymmKey, by general you mean the cluster pass4SymmKey??

0 Karma

Steve_G_
Splunk Employee
Splunk Employee

You must set the pass4SymmKey on the forwarders when using indexer discovery. The docs were incorrect on this issue, but have now been updated. See http://docs.splunk.com/Documentation/Splunk/6.3.0/Indexer/indexerdiscovery#3._Configure_the_forwarde...

0 Karma

Steve_G_
Splunk Employee
Splunk Employee

Also, if you do not explicitly set pass4SymmKey in the cluster master's [indexer_discovery] stanza, the master will use the value in its [general] stanza - either a value that you have explicitly set there or the default value.

In either case, the forwarder's value in [indexer_discovery] must match that value.

Therefore, the simplest way to deal with this is to set pass4SymmKey in the [indexer_discovery] stanza on the master, as well as on all the forwarders.

0 Karma

asimagu
Builder

hum...thanks for your help, but I think there is something else going on.
I copied the exact config from the doc example on my CM and on my HWF. As soon as I restart the HWF with the new config, it even stops indexing internal stuff so I have to manually go to splunkd.log and this is the ERROR that I get:

10-02-2015 10:52:43.908 +0100 ERROR IndexerDiscoveryHeartbeatThread - failed to parse response payload for group=group1, err=failed to extract FwdTarget from json node={"hostport":"?","ssl":false,"indexing_disk_space":23184490496}http_response=OK

any ideas? it seems there is something funky going on with the IndexerDiscovery but at the same time I don't get why the HWF stops indexing its own internal stuff

0 Karma

Lowell
Super Champion

I had this same error, but in my case it was caused by something weird with one of the indexers. I ran the CURL command noted above and saw that one of the 4 indexes was missing (or "replaced with hostport="?"). So I went to the "missing" indexer, ran a splunk offline and now the UF started working correctly and the CURL command returns only a valid list of indexers. Very weird.

0 Karma

Lowell
Super Champion

For anyone who cares (or next time I run into this issue), ... I found and resolved my issue.

I missed setting up a TCP Input on one of the 4 peer nodes. (This is why automation rocks, and doing stuff by hand is evil).

I found that if I hit the /services/cluster/config endpoint on all the peer nodes, the one that was causing issues was returning ? for forwarderdata_rcv_port. Apparently the cluster master sends this bogus value to the UF's via the /services/indexer_discovery endpoint. Whoops.

Find it in splunk like this:

| rest services/cluster/config | search mode=slave forwarderdata_rcv_port="?"

Lucas_K
Motivator

I got a similar issue. I was using the master to give itself a list of peers to send internal logs to however and not a heavy forwarder.

In this case I HAD use the password as listed in the [general] stanza and NOT the indexer_discovery one. Even though both could be set the indexer discovery one even when set with plaintext will fail.

I was getting the exact same error and have just resolved it in the past few minutes.

0 Karma

asimagu
Builder

thanks for your input Lucas, I ended up doing the exact same thing, so I guess that either the documentation is wrong or this is a bug and a possible workaround?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...