FAILED_LOAD_DEPLOYMENT_SERVER__TENANT_default

[ShaneNewman]

I am having an issue with an indexer, this is the information from the logs:

Any help would be greatly appreciated.

11-26-2013 14:08:09.796 -0600 WARN  PropertiesMap - Cannot convert string "flase" into bool for "SHOULD_LINEMERGE"
11-26-2013 14:08:12.031 -0600 ERROR AdminManager - Stack trace from python handler:\nTraceback (most recent call last):\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\admin.py", line 70, in init\n    hand.execute(info)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\admin.py", line 527, in execute\n    if self.requestedAction == ACTION_LIST:     self.handleList(confInfo)\n  File "C:\Program Files\Splunk\etc\system\bin\DataModelAccelerationHandler.py", line 20, in handleList\n    sc_rest.BaseRestHandler.handleList(self, confInfo)\n  File "C:\Program Files\Splunk\etc\system\bin\sc_rest.py", line 74, in handleList\n    ent = self.all()\n  File "C:\Program Files\Splunk\etc\system\bin\sc_rest.py", line 221, in all\n    offset=self.posOffset)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\entity.py", line 129, in getEntities\n    atomFeed = _getEntitiesAtomFeed(entityPath, namespace, owner, search, count, offset, sort_key, sort_dir, sessionKey, uri, hostPath, **kwargs)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\entity.py", line 222, in _getEntitiesAtomFeed\n    serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\rest\__init__.py", line 461, in simpleRequest\n    serverResponse, serverContent = h.request(uri, method, headers=headers, body=payload)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\httplib2\__init__.py", line 1421, in request\n    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\httplib2\__init__.py", line 1171, in _request\n    (response, content) = self._conn_request(conn, request_uri, method, body, headers)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\httplib2\__init__.py", line 1147, in _conn_request\n    response = conn.getresponse()\n  File "C:\Program Files\Splunk\Python-2.7\Lib\httplib.py", line 1033, in getresponse\n    raise ResponseNotReady()\nResponseNotReady\n
11-26-2013 14:08:12.031 -0600 ERROR AdminManager - Unexpected error "<class 'httplib.ResponseNotReady'>" from python handler: "".  See splunkd.log for more details.
11-26-2013 14:08:12.031 -0600 ERROR SummarizationHandler - Error listing accelerated data models: Unexpected error "<class 'httplib.ResponseNotReady'>" from python handler: "".  See splunkd.log for more details.
11-26-2013 14:08:12.140 -0600 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=rejected
11-26-2013 14:08:24.140 -0600 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=rejected
11-26-2013 14:08:36.140 -0600 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=rejected
11-26-2013 14:08:39.406 -0600 INFO  databasePartitionPolicy - idx=os_aix Creating hot bucket=hot_v1_30, given event timestamped=1385496510
11-26-2013 14:08:39.406 -0600 INFO  DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='C:\Program Files\Splunk\var\lib\splunk\os_aix\db'.  Reason='Bucket directory structure changed.'
11-26-2013 14:08:44.515 -0600 INFO  KeyManagerSearchPeers - Reading public key for peer: C:\Program Files\Splunk\etc\auth\distServerKeys\NADCWPAPPSPL01\trusted.pem
11-26-2013 14:08:44.515 -0600 INFO  KeyManagerSearchPeers - Finished reading public key for peer: C:\Program Files\Splunk\etc\auth\distServerKeys\NADCWPAPPSPL01\trusted.pem
11-26-2013 14:08:44.968 -0600 ERROR AdminManager - Stack trace from python handler:\nTraceback (most recent call last):\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\admin.py", line 70, in init\n    hand.execute(info)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\admin.py", line 527, in execute\n    if self.requestedAction == ACTION_LIST:     self.handleList(confInfo)\n  File "C:\Program Files\Splunk\etc\system\bin\DataModelAccelerationHandler.py", line 20, in handleList\n    sc_rest.BaseRestHandler.handleList(self, confInfo)\n  File "C:\Program Files\Splunk\etc\system\bin\sc_rest.py", line 74, in handleList\n    ent = self.all()\n  File "C:\Program Files\Splunk\etc\system\bin\sc_rest.py", line 221, in all\n    offset=self.posOffset)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\entity.py", line 129, in getEntities\n    atomFeed = _getEntitiesAtomFeed(entityPath, namespace, owner, search, count, offset, sort_key, sort_dir, sessionKey, uri, hostPath, **kwargs)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\entity.py", line 222, in _getEntitiesAtomFeed\n    serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\rest\__init__.py", line 461, in simpleRequest\n    serverResponse, serverContent = h.request(uri, method, headers=headers, body=payload)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\httplib2\__init__.py", line 1421, in request\n    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\httplib2\__init__.py", line 1171, in _request\n    (response, content) = self._conn_request(conn, request_uri, method, body, headers)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\httplib2\__init__.py", line 1147, in _conn_request\n    response = conn.getresponse()\n  File "C:\Program Files\Splunk\Python-2.7\Lib\httplib.py", line 1033, in getresponse\n    raise ResponseNotReady()\nResponseNotReady\n
11-26-2013 14:08:44.968 -0600 ERROR AdminManager - Unexpected error "<class 'httplib.ResponseNotReady'>" from python handler: "".  See splunkd.log for more details.
11-26-2013 14:08:44.968 -0600 ERROR SummarizationHandler - Error listing accelerated data models: Unexpected error "<class 'httplib.ResponseNotReady'>" from python handler: "".  See splunkd.log for more details.
11-26-2013 14:08:48.140 -0600 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=rejected
11-26-2013 14:09:00.140 -0600 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=rejected
11-26-2013 14:09:00.359 -0600 WARN  PubSubConnection - Cannot convert str:  to a valid status, returning eRejected.
11-26-2013 14:09:04.125 -0600 ERROR AdminManager - Stack trace from python handler:\nTraceback (most recent call last):\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\admin.py", line 70, in init\n    hand.execute(info)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\admin.py", line 527, in execute\n    if self.requestedAction == ACTION_LIST:     self.handleList(confInfo)\n  File "C:\Program Files\Splunk\etc\system\bin\DataModelAccelerationHandler.py", line 20, in handleList\n    sc_rest.BaseRestHandler.handleList(self, confInfo)\n  File "C:\Program Files\Splunk\etc\system\bin\sc_rest.py", line 74, in handleList\n    ent = self.all()\n  File "C:\Program Files\Splunk\etc\system\bin\sc_rest.py", line 221, in all\n    offset=self.posOffset)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\entity.py", line 129, in getEntities\n    atomFeed = _getEntitiesAtomFeed(entityPath, namespace, owner, search, count, offset, sort_key, sort_dir, sessionKey, uri, hostPath, **kwargs)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\entity.py", line 222, in _getEntitiesAtomFeed\n    serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\rest\__init__.py", line 461, in simpleRequest\n    serverResponse, serverContent = h.request(uri, method, headers=headers, body=payload)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\httplib2\__init__.py", line 1421, in request\n    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\httplib2\__init__.py", line 1171, in _request\n    (response, content) = self._conn_request(conn, request_uri, method, body, headers)\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\httplib2\__init__.py", line 1147, in _conn_request\n    response = conn.getresponse()\n  File "C:\Program Files\Splunk\Python-2.7\Lib\httplib.py", line 1033, in getresponse\n    raise ResponseNotReady()\nResponseNotReady\n
11-26-2013 14:09:04.125 -0600 ERROR AdminManager - Unexpected error "<class 'httplib.ResponseNotReady'>" from python handler: "".  See splunkd.log for more details.
11-26-2013 14:09:04.125 -0600 ERROR SummarizationHandler - Error listing accelerated data models: Unexpected error "<class 'httplib.ResponseNotReady'>" from python handler: "".  See splunkd.log for more details.
11-26-2013 14:09:12.140 -0600 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=rejected
11-26-2013 14:09:24.140 -0600 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=rejected

[ShaneNewman · Accepted solution]

There is a bug in Splunk 6.0.0.2 pertaining to WS2003R2. There has been a bug ticket created and they should have more information by 12/3/2013. The bug apparently corrupts the registry and keeps splunkweb from installing.

The only way to resolve this was to delete the services manually, then reinstall Splunk 5.0.5. Splunkweb will still not load, it does give the ability to at least get the services up and running though.