All Apps and Add-ons

has anyone successful setup the remotePath option in indexes.conf in Splunk 7.0 to work with indexed data in s3?

Log_wrangler
Builder

I was trying to search copies of indexed data in S3.

Has anyone had luck with this scenario using remotePath??? I know it says not supported but is it functional at this point?

indexes.conf for splunk 7.0
[https://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/Indexesconf]

remotePath =
* Currently not supported. This setting is related to a feature that is
still under development.
* Optional.
* Presence of this parameter means that this index uses remote storage, instead
of the local file system, as the main repository for bucket storage. The
index processor works with a cache manager to fetch buckets locally, as
necessary, for searching and to evict them from local storage as space fills
up and they are no longer needed for searching.
* This setting must be defined in terms of a storageType=remote volume
definition. See the volume section below.
* The path portion that follows the volume reference is relative to the path
specified for the volume. For example, if the path for a volume "v1" is
"s3://bucket/path" and "remotePath" is "volume:v1/idx1", then the fully
qualified path will be "s3://bucket/path/idx1". The rules for resolving the
relative path with the absolute path specified in the volume can vary
depending on the underlying storage type.
* If "remotePath" is specified, the "coldPath" and "thawedPath" attributes are
ignored. However, they still must be specified.

Any advise or lessons learned is appreciated.

Thank you

0 Karma

nickhills
Ultra Champion
remotePath = <root path for remote volume, prefixed by a URI-like scheme>

Currently not supported. This setting is related to a feature that is
still under development.

Even if i knew, I couldn't tell you!
:)

My guess is that its to allow for cloud storage like S3 - hopefully for archive/frozen data - but thats entirely a guess
But i am watching the releases very closely for more info

If my comment helps, please give it a thumbs up!

nickhills
Ultra Champion

Just a follow up comment - I very much doubt that this is going to allow you to store hot/warm or even cold data in s3 - the read performance isn't up to the job, and the write mechanism would be far too clunky for indexers.

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...