All Apps and Add-ons

Has anyone attempted to run Splunk with indexes stored on AWS Elastic File System (EFS)?

swilsonGresham
Explorer

Has anybody attempted to run Splunk with the indexes stored on AWS Elastic File System (EFS)?

Reading up on EFS, its seems ideally designed for a tool such as Splunk and, since it is elastic, means you can easily eliminate any disk space issues.

Specifically the AWS site says:

It’s designed for high availability and durability, and provides performance for a broad spectrum of workloads and applications, including Big Data and analytics, media processing workflows, content management, web serving, and home directories.

I am interested to hear views on using it and whether anybody has attempted to use it?

sarakirm
New Member

@treinke Thank you for this info. Did you ever get a reply from support? I'm wondering whether your diagnosis was confirmed, and whether you found a way around it?

0 Karma

treinke
Builder

@sarakirm - I was told that EFS was not a supported target drive space and should look at using a supported drive.

There are no answer without questions
0 Karma

vguptadevops
New Member

Hello @swilsonGresham I just came across the question when I was looking for the very similar information. As of right now I have designed my splunk in AWS to just use EBS with regular IOPS , I expect around 50 GB/day data , I just thought to use EFS only today but I wanted to see how the community and people's experience was there as @dwaddle pointed out NFS v4 which provides the good ssd with little low IOPS level performance , I am not sure what would my splunk do , so if you have used EFS and would like to share the thoughts It would be awesome.

Thanks much

0 Karma

dwaddle
SplunkTrust
SplunkTrust

EFS uses NFS v4 to mount on Linux hosts. Splunk generally does not recommend the use of NFS for hot buckets. See http://docs.splunk.com/Documentation/Splunk/6.4.3/Installation/Systemrequirements#Considerations_reg...

It might do well for cold buckets. But even then I'm not really sure.

From Amazon's FAQs on the product:

All file systems deliver a consistent baseline performance of 50 MB/s per TB of storage, all file systems
(regardless of size) can burst to 100 MB/s, and file systems larger than 1TB can burst to 100 MB/s per TB of storage.

When you have multiple Splunk nodes trying to hit an EFS volume in parallel and you have an aggregate of 100MB/sec/TB across all of your indexers - you would need to do the math to be sure that given the size of your volume and number of indexers that you don't create a bottleneck.

dwaddle
SplunkTrust
SplunkTrust

December 2018 update: Some folks have tried to use EFS for certain Splunk architectures and it has not gone well. As @treinke discovered, use of EFS combined with Splunk Clustering has some distinct issues that make the cluster non-operative. Proceed with caution.

treinke
Builder

The problem with EFS is that it is mounted as a single large drive. The size? 8.0E. What I was seeing was that the available size was large enough to cause a negative wrap on the value. This was causing Splunk to show the available size as a negative value and causing Splunk to shutdown do to the available storage is too low. I opened a support ticket (950970) on this issue.

There are no answer without questions
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...