search peer searching requires invocation of splunk_server=* and indexes not available in roles

[leonphelps_s]

Seems like a relatively simple issue but I'm stumped.

I've got peers setup on on a search head.. and if I do a search referencing an index ONLY available on the remote peer.. it will only work if I do it as follows.

index=INDEX_NAME splunk_server=*

if I don't invoke splunk_server= no results are returned.

Further, when in access control none of the indexes are shown from the peer.

By default only the main index is searched. I did try setting default searched indexes to all non_internal but that didn't work either unless i invoked splunk_server.

What am I missing?

[leonphelps_s · Accepted solution]

This appears to be a result of a bug .."(SPL-95114) where instability in an indexing cluster can lead to distributed search groups losing the internal reference to their members. When this happens, the default target group "dmc_group_indexer" acts as if it had no members even though those are listed in distsearch.conf, and as a result your searches will not be dispatched to any remote peers unless you specify a splunk_server=* or splunk_server_group=* clause."
https://answers.splunk.com/answers/221468/search-returns-zero-results-searchlog-reports-dist.html
Cleaning up distsearch.conf with splunk restart resolved this issue

[Lucas_K]

Check your distsearch.conf on the search head.

Do you have anything odd in it?

[leonphelps_s]

servers = localhost:localhost

[distributedSearch:dmc_group_search_head]
servers = localhost:localhost

[distributedSearch:dmc_group_cluster_master]

[distributedSearch:dmc_group_deployment_server]

[distributedSearch:dmc_group_kv_store]

[distributedSearch:dmc_group_indexer]
default = true
servers = localhost:localhost

[distributedSearch:dmc_group_shc_deployer]

[distributedSearch]
servers = https://name_removed.com:8089

so basically nothing but the peer info

[leonphelps_s]

Looks like the stuff was "crazy" lol will answer my original question with the fix.