Deployment Architecture

search peer searching requires invocation of splunk_server=* and indexes not available in roles

leonphelps_s
Path Finder

Seems like a relatively simple issue but I'm stumped.

I've got peers setup on on a search head.. and if I do a search referencing an index ONLY available on the remote peer.. it will only work if I do it as follows.

index=INDEX_NAME splunk_server=*

if I don't invoke splunk_server= no results are returned.

Further, when in access control none of the indexes are shown from the peer.

By default only the main index is searched. I did try setting default searched indexes to all non_internal but that didn't work either unless i invoked splunk_server.

What am I missing?

Tags (1)
0 Karma
1 Solution

leonphelps_s
Path Finder

This appears to be a result of a bug .."(SPL-95114) where instability in an indexing cluster can lead to distributed search groups losing the internal reference to their members. When this happens, the default target group "dmc_group_indexer" acts as if it had no members even though those are listed in distsearch.conf, and as a result your searches will not be dispatched to any remote peers unless you specify a splunk_server=* or splunk_server_group=* clause."
https://answers.splunk.com/answers/221468/search-returns-zero-results-searchlog-reports-dist.html
Cleaning up distsearch.conf with splunk restart resolved this issue

View solution in original post

0 Karma

leonphelps_s
Path Finder

This appears to be a result of a bug .."(SPL-95114) where instability in an indexing cluster can lead to distributed search groups losing the internal reference to their members. When this happens, the default target group "dmc_group_indexer" acts as if it had no members even though those are listed in distsearch.conf, and as a result your searches will not be dispatched to any remote peers unless you specify a splunk_server=* or splunk_server_group=* clause."
https://answers.splunk.com/answers/221468/search-returns-zero-results-searchlog-reports-dist.html
Cleaning up distsearch.conf with splunk restart resolved this issue

0 Karma

Lucas_K
Motivator

Check your distsearch.conf on the search head.

Do you have anything odd in it?

0 Karma

leonphelps_s
Path Finder

servers = localhost:localhost

[distributedSearch:dmc_group_search_head]
servers = localhost:localhost

[distributedSearch:dmc_group_cluster_master]

[distributedSearch:dmc_group_deployment_server]

[distributedSearch:dmc_group_kv_store]

[distributedSearch:dmc_group_indexer]
default = true
servers = localhost:localhost

[distributedSearch:dmc_group_shc_deployer]

[distributedSearch]
servers = https://name_removed.com:8089

so basically nothing but the peer info

0 Karma

leonphelps_s
Path Finder

Looks like the stuff was "crazy" lol will answer my original question with the fix.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...