Splunk Search

Initiating Splunk on AWS AMI, why am I getting "Search not executed: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch. user=admin"

gmark
Explorer

I've initiated an AMI of Splunk on a t2.medium instance, and even before I've actively used it, I get

Search not executed: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch. user=admin 

yet removing files from that path has no effect, and the files there look pretty small.

Any ideas? Will I need to allocate more HD already, even though I haven't even used the thing? Are AMI instances of Splunk very different from those running on physical servers?

0 Karma

pwmcity
Path Finder

If you actually want to use this, you'll need to allocate more disk.
I recommend using an EBS, not the ephemeral disk that comes attached by default.
Once attached / formatted / mounted, install / move your splunk instance to wherever your EBS is mounted.

If you don't care about any of that and are okay with potentially losing your data / config... then just up the limit, see here:
http://answers.splunk.com/answers/214804/the-minimum-free-disk-space-5000mb-reached-for-opt.html

0 Karma

gmark
Explorer

I do want to know if this is normal or if something is wrong. Since this is the initial installation as provided by the one-click method I was given on the AWS website, I can't understand why that configuration isn't sufficient to use Splunk. Any idea?

0 Karma

gmark
Explorer

Wow - I see the specs on the default AMI are "1.0GB main memory and 0GB storage / EBS only" I'd assumed
that this meant that there would be some storage available that would simply not survive termination of the instance.

It seems kind of silly, but does this ACTUALLY mean that there is NO ROOM for indexes at all?

How could ANY instance of Splunk EVER work on this? Am I reading this wrong?

0 Karma

pwmcity
Path Finder

Sounds like there is an EBS, but maybe something went wrong, didn't attach, and so the Splunk installation defaulted to the ephemeral storage ... which can happen! Maybe try again?

0 Karma

pwmcity
Path Finder

oh the marketplace AMI? I looked at that once and based on how old it is and the terrible reviews, decided to build my own. Considering it's their official AMI, they sure have done a terrible job maintaining it.

How's your linux / aws foo? Can you SSH onto the box and run a 'df -h' to check disk space, and a 'mount -l' to see your mounts

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...