All Apps and Add-ons

Splunk Add-on for AWS Hangs

grahmorl
Explorer

Hi,

I'm having issues getting the Splunk Add-on for AWS to work on a completely clean installation of Splunk.

I'm using the latest version of the Add-on from Splunkbase (v4.4.0) and I've tried on both Splunk 6.6.6 and 7.0.2

The problem I'm having is that if you go to either the 'Inputs' or 'Configuration' pages, the web interface just hangs with a spinning 'Loading' icon.

This is similar to this question: https://answers.splunk.com/answers/338274/splunk-app-for-aws-configuration-page-hangs.html

But unfortunately there's no accepted answer for this.

Looking in splunkd.log

index=_internal source="/opt/splunk/var/log/splunk/splunkd.log" sourcetype=splunkd ERROR aws

It appears to be a whole host of REST errors.

Has anyone else come across this issue? And perhaps has a solution?

Thanks.

1 Solution

grahmorl
Explorer

After a few months of back and forth with Splunk Support, we got to the root cause.

The credit for this goes to the unnamed (sorry, I don't know their name) AWS Add-on developer, who the support case got escalated to. They flagged that there was a global boto configuration file (/etc/boto.cfg) which was causing a conflict with some assumed defaults which the AWS Add-on relies on.

To summarise:

Problem:
- This problem presented as a hanging / spinning ‘Loading’ icon in the Splunk AWS Add-on Inputs and Configuration views.
- A search in the _internal index for “aws ERROR” showed python stack trace error messages.
- The problem appears to be configuration in the /etc/boto.cfg file. This is a global configuration file, so the settings in here are conflicting with the defaults (if there was no config file), which the Splunk AWS Add-on relies on.
- The file is created due to the default installation of Google’s gcloud / gsutil command line tools. The tools are installed by default on Google Cloud Compute (GCP) hosted servers.
- Our initial AWS integration testing was using GCP account to host the HFs, whilst waiting for access to a different AWS account.

Solution:
- You don't really want remove the /etc/boto.cfg configuration file because those servers might need these tools to access GCP services / APIs.
- In addition, even if we removed it, there is the possibility that other tools could create it in the future.
- Therefore the fix is to do something which will only affect Splunk.
- What we did was, in the $SPLUNK_HOME/etc/splunk-launch.conf file we added the environment variable:

BOTO_CONFIG=/tmp/does_not_exist

- This sets the BOTO_CONFIG variable for just the Splunk process specifically to a file which doesn’t exist.
- Because the files doesn’t exist, the defaults, as opposed to the global /etc/boto.cfg are used.
- Therefore there is now no conflict for the Splunk AWS Add-on, which runs under the Splunk process.

I hope this helps others who may be having similar issues. Or at least points them in a direction to look into.

Graham.

View solution in original post

grahmorl
Explorer

After a few months of back and forth with Splunk Support, we got to the root cause.

The credit for this goes to the unnamed (sorry, I don't know their name) AWS Add-on developer, who the support case got escalated to. They flagged that there was a global boto configuration file (/etc/boto.cfg) which was causing a conflict with some assumed defaults which the AWS Add-on relies on.

To summarise:

Problem:
- This problem presented as a hanging / spinning ‘Loading’ icon in the Splunk AWS Add-on Inputs and Configuration views.
- A search in the _internal index for “aws ERROR” showed python stack trace error messages.
- The problem appears to be configuration in the /etc/boto.cfg file. This is a global configuration file, so the settings in here are conflicting with the defaults (if there was no config file), which the Splunk AWS Add-on relies on.
- The file is created due to the default installation of Google’s gcloud / gsutil command line tools. The tools are installed by default on Google Cloud Compute (GCP) hosted servers.
- Our initial AWS integration testing was using GCP account to host the HFs, whilst waiting for access to a different AWS account.

Solution:
- You don't really want remove the /etc/boto.cfg configuration file because those servers might need these tools to access GCP services / APIs.
- In addition, even if we removed it, there is the possibility that other tools could create it in the future.
- Therefore the fix is to do something which will only affect Splunk.
- What we did was, in the $SPLUNK_HOME/etc/splunk-launch.conf file we added the environment variable:

BOTO_CONFIG=/tmp/does_not_exist

- This sets the BOTO_CONFIG variable for just the Splunk process specifically to a file which doesn’t exist.
- Because the files doesn’t exist, the defaults, as opposed to the global /etc/boto.cfg are used.
- Therefore there is now no conflict for the Splunk AWS Add-on, which runs under the Splunk process.

I hope this helps others who may be having similar issues. Or at least points them in a direction to look into.

Graham.

splunkcol
Builder

 

this has fixed the problem, thanks

0 Karma

girishkalamati
Observer

This did not fix my issue

0 Karma

richgalloway
SplunkTrust
SplunkTrust

@grahmorl If your problem is resolved, please accept an answer to help future readers.

---
If this reply helps you, Karma would be appreciated.
0 Karma

grahmorl
Explorer

@richgalloway - Of course. I was just waiting to my answer to be approved by the moderator. 😉

0 Karma

klaxdal
Contributor

I encountered a similar issue - removed everything in the /local - restarted splunk and it stopped hanging .

From there reconfigured my inputs and all was well -

splunkcol
Builder

You saved me, at first I applied the process mentioned in this thread, but today for some reason the problem returned

I ran the command in linux df -h and a folder was full.

After deleting junk information and restarting Splunk I have solved it

Thank you

0 Karma

girishkalamati
Observer

what does ' /local ' means , I can go to this location '$SPLUNK_HOME/etc/' though. Thanks if you can explain how I can reach to /local and empty files if anything persist so that I can fix this loading issue.

Is the also a possible reason : SSL certificatse ?

0 Karma

grahmorl
Explorer

@klaxdal - My issue was actually happening on a completely new install of Splunk and the AWS components. So nothing configured or in /local

So I think my problem was a little different to yours.

But many thanks for your comment.

0 Karma

deepashri_123
Motivator

Hey grahmorl,

What is the hardware specification you are using? Are you using EBS volumes or are you using local storage?

0 Karma

grahmorl
Explorer

@deepashri_123 - Thanks for your comment. The issue was somewhat more involved and not hardware related (i.e. not a performance issue). But thank your message.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...