Role/User default search indexes not working

[yaleman]

I can't tell if it's since we moved to mounted bundles or not, but recently we need to explicitly set the indexes which we wish to search. It was working perfectly not long ago. Doing a search of just * gives not much at all - only main, _internal, _audit, _introspection and sos. There's another ~15 indexes with MANY more events in them on our system.

I've gone as far as clicking "add all" on the "default indexes searched" box and it doesn't seem to make any difference. This is breaking all our apps, amongst other issues.

We've got a single search head (v6.1.x) pointing at a single indexer (v6.0.x) both running Debian with the indexer NFS-mounting the search head's /opt/splunk/etc/ directory directly.

[Lucas_K]

Check distributed search on the search head (settings/distributed search/search peers). Do you see your index listed correctly as "UP"?

Do you have your mount point correctly configured on the indexer? The incoming search from the sh to the idx needs this so it can correctly get app configurations.

on your indexer check your distsearch.conf - /opt/splunk/etc/system/local/distsearch.conf

[my_searchhead]
mounted_bundles=true
bundles_location=/mnt/shared_bundle

Check your auth on the search head.
You can use btool to show this.
splunk btool authorize list --debug

Find your role does it have the correct indexes listed?

If not fix them in /opt/splunk/etc/system/local/authorize.conf on your search head.

[Lucas_K]

Did the btool output look the same as what your expecting to see?

My comment about the bundle was that you've exported /opt/splunk/etc directly. Nothing in the docs said to do that, it should work as the indexer should still be able to figure out where the app artifacts are. I don't 100% know if it picks up anything else that it shouldn't be seeing. My mounts consist of apps/pooling/system and users dirs only nothing else (but that is shared storage not the search heads etc dir).

[yaleman]

Apologies, yes. The index is showing status of "up" and replication status of "mounted". Searches work fine (with no errors) as long as we specify the index manually in the search string.

I'm not sure what you mean by the fact that it's abnormal - it's configured exactly as per the docs?

[Lucas_K]

You didn't mention if you can see the index from the search head (settings/distributed search/search peers).

Also, tried switching back to non-mounted bundles? Your bundling setup seems abnormal compared to the documentation.

[yaleman]

The distsearch.conf seems to be OK, matches up with the mounted directory. My account's got two roles (admin and can_delete) - which it always has had.

The authorize.conf seems sane - admin has a long list of srchIndexesDefault (including *) - and can_delete doesn't have a line for that. I'm stumped.