Multitenant environments, and creating indexes on the fly

[sideview]

I'm interested in setting up a Splunk server where each customer would have their own indexes and would only be able to search that one index.

However we'd definitely need to build the overall system such that the indexes, users, roles could be created on the fly without restarting the server.

I know you cannot create indexes on the fly in 4.1 (ie without restarting), and although the docs don't say that you can do this in 4.2, I thought I'd ask -- can this be done in 4.2 and if so how would you go about it?

[hexx · Accepted solution]

This is indeed a new 4.2 functionality. From the CLI, one can simply run :

$SPLUNK_HOME/bin/splunk add <index_name>

$SPLUNK_HOME/bin/splunk reload index

The newly-created index will be ready to use at this point.

The UI will do the "splunk reload index" for you, so any indexes created from the UI are ready to be used right away.

One caveat here is that on a system where splunkd is very busy, there might be a delay between the execution of the index reload and the actual availability of the index to throw data at.

As a rule of thumb, this delay is usually measured in seconds, but if you want to be sure that the index will be available, I would recommend to wait for 30 seconds to 1 minute after the dynamic reload before sending data to it.

Do note that "splunk reload index" only allows to add new indexes. You will not be able to remove a pre-existing index on the fly in this way.

[rmorlen]

Figured it out. We had an issue with the indexes.conf (duplicate index). Fixed it and restested. All worked great!

curl -k -u admin:changeme https://localhost:8089/servicesNS/admin/indexerbase/configs/conf-indexes/_reload

then
curl -k -u admin:changeme https://localhost:8089/services/data/indexes/_reload

[hexx]

That is not entirely surprising, I am afraid that endpoint only supports the addition of newly-defined indexes. You can't dynamically reload the configuration when removing or modifying an existing index.

[rmorlen]

Error. In handler 'indexes': reload is not safe since a path has been deleted or modified for an index, or an index has been disabled. You must restart the Splunk Server, for your changes to take effect.

[hexx]

@rmorlen: Not if you're pushing apps. Or you'd have to complement your app push with a REST call to the /services/data/indexes/_reload endpoint to ask for a dynamic reload of the indexes configuration.

[rmorlen]

Our indexes are all configured using an app. That app then gets pushed to all of our indexers and the index entry gets created. We still have to bounce the indexers to get the index file system created. Any better way of handling this?

[hexx]

In a distributed deployment environment, you would have to use the CLI remotely (with a -uri option) or direct REST API calls to create new indexes and reload the indexing configuration on remote indexers. And yes, this operation (index creation + conf reload) would need to be performed on each indexer.

[tpsplunk]

can you comment on how to use this with a distributed deployment? can i update the indexes.conf file manually or do i have to use the splunk add command?
i presume i'd have to run the splunk reload index command on each indexer?