Splunk_TA_New_Relic internal server error

[mamiller400]

I'm seeing internal server errors returned from the Splunk add on for New Relic

04-03-2017 16:37:04.992 -0400 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 129, in init\n    hand.execute(info)\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 590, in execute\n    if self.requestedAction == ACTION_LIST:     self.handleList(confInfo)\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/splunk_aoblib/rest_migration.py", line 38, in handleList\n    AdminExternalHandler.handleList(self, confInfo)\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/splunktaucclib/rest_handler/admin_external.py", line 40, in wrapper\n    for entity in result:\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/splunktaucclib/rest_handler/handler.py", line 120, in wrapper\n    raise RestError(500, traceback.format_exc())\nRestError: REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/splunktaucclib/rest_handler/handler.py", line 113, in wrapper\n    for name, data, acl in meth(self, *args, **kwargs):\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/splunktaucclib/rest_handler/handler.py", line 299, in _format_response\n    masked = self.rest_credentials.decrypt_for_get(name, data)\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/splunktaucclib/rest_handler/credentials.py", line 184, in decrypt_for_get\n    clear_password = self._get(name)\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/splunktaucclib/rest_handler/credentials.py", line 388, in _get\n    string = mgr.get_password(user=context.username())\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/solnlib/utils.py", line 150, in wrapper\n    return func(*args, **kwargs)\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/solnlib/credentials.py", line 118, in get_password\n    all_passwords = self._get_all_passwords()\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/solnlib/utils.py", line 150, in wrapper\n    return func(*args, **kwargs)\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/solnlib/credentials.py", line 232, in _get_all_passwords\n    all_passwords = self._storage_passwords.list(count=-1)\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/solnlib/packages/splunklib/client.py", line 1459, in list\n    return list(self.iter(count=count, **kwargs))\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/solnlib/packages/splunklib/client.py", line 1419, in iter\n    items = self._load_list(response)\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/solnlib/packages/splunklib/client.py", line 1325, in _load_list\n    entries = _load_atom_entries(response)\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/solnlib/packages/splunklib/client.py", line 201, in _load_atom_entries\n    r = _load_atom(response)\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/solnlib/packages/splunklib/client.py", line 196, in _load_atom\n    return data.load(response.body.read(), match)\n  File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/solnlib/packages/splunklib/data.py", line 77, in load\n    root = XML(text)\n  File "/opt/splunk/lib/python2.7/xml/etree/ElementTree.py", line 1300, in XML\n    parser.feed(text)\n  File "/opt/splunk/lib/python2.7/xml/etree/ElementTree.py", line 1642, in feed\n    self._raiseerror(v)\n  File "/opt/splunk/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror\n    raise err\nParseError: not well-formed (invalid token): line 237, column 37\n\n

It seems related to authentication based on the error, however I cannot even load the Configuration tabs to set up proxy info. I can't find configuration examples on how to make these changes from config files either.
EDIT: some additional info
We're running Splunk 6.5.2 in a distributed environment on all Ubuntu Server 16.04 nodes - the add-on has been installed on the indexers as well as the stand alone search head I'm using to test, the Splunk app for New Relic has been installed on the stand alone search head. I have obviously not tested the app as the add on is not working.

[kkrishnan_splun]

Has anyone found the answer to this ? I believe I have the same issue

[HMTODD]

You should try the new version 2 of the Splunk Add-on for New Relic. That should work better. Also, be sure to test New Relic access using the New Relic REST API Explorer.

[mamiller400]

I installed the newer version but am still experiencing the same internal server error as before.

[mamiller400]

It is a stand alone search head, clustered indexers (they do have the TA installed) and the above error is from splunkd.log when trying to open the configuration tab. In the UI, there is just a spinning wheel that never stops, no error is displayed.

[ehaddad_splunk]

• is this a clustered environment?
• Did you setup/configure the TA?
• When you click on the Add-on, do you see the "configuration" tab? If config tab doesn't load whats the error?