Installation

Errors when upgrading from 5.0.4 to 6.1.3.

immortalraghava
Path Finder

Hi,

We are upgrading from 5.0.4 to 6.1.3 in a distributed Splunk environment. There are multiple instances of Splunk running as Forwarder, Indexer, and single Deployment server(License Master) which also runs some scheduled searches and sends data back to Indexers. In our upgrade process we stopped Splunk process in the node extract Splunk 6.1.3 on the existing Splunk installation path and again started Splunk with "Splunk start accept license" command. We do this node by node starting with Indexer, forwarder, and deployment server. Now after installation is complete in all nodes, the deployment server splunkd.log contains the following error messages

GetRemoteAuthToken - Unable to get authentication token from peeruri="https://INDEXER:8089/services/admin/auth-tokens

Unable to distribute to peer named INDEXER at uri https://<Indexer>:8089 because peer has status = "Authentication Failed"

The following procedure resolves this error
Remove the indexer entry from distsearch.conf file and restart Splunk.

[distributedSearch]
servers = INDEXER:8089

After that again add the same entry in the same file and restart Splunk.

This happens only after the upgrade.
Any help is appreciated.
Thanks!

Labels (4)

davebo1896
Communicator

I stumbled across this when upgrading to 6.3.0
I think it is related to the distServerKeys being regenerated.
Make sure to save your private.pem and trusted.pem as well as the remote keys from other servers and put them back into etc/auth/distServerKeys/.

jkat54
SplunkTrust
SplunkTrust

each indexer has a $SPLUNK_HOME/etc/auth/distServerKeys/{SEARCH_HEAD}/trusted.pem for each search head and the cluster master.

If for whatever reason your indexer "knew" of your search head in the past, and you've recently rebuilt your search head or created a clone with the same servername, etc. then you may have to delete the {SEARCH_HEAD} dir show above on each indexer that previously knew of the search head. Then, add the peer again via the UI on the search head.

It happened to us when we changed from single search head to search head cluster, and we re-used the old search head in the new SHC.

lakshman237
Path Finder

Hopefully this resolved by now. This usually appears when the search head cannot authenticate with search peers in the distributed environment. Pls re-authenticate either using CLI or from the GUI within the search head : settings-> distributed search -> Search peers and ensure, its up and enabled.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...