Deployment Architecture

Search head returning no results from peer

TAE2112
Explorer

I have a fairly basic deployment - one Search Head configured with two distributed search peers/indexers. Each peer is in a different data center, serving as the role of an indexer. I've installed the Universal Forwarder on the hosts in each datacenter, and they are successfully sending Windows Event Log data to their respective indexer. No clustering or replication is taking place, and both indexers have checked into the licensing service (which is also running on my Search Head). Both peers are showing up as "Up" and replication status is "Successful".

On each indexer, I can execute searches against indexes specific to the indexes hosted on each indexer. Put another way (because that sounded really confusing), when I search for data appropriate to each indexer, I receive the expected results.

While logged into the search head, when I search for data specific to data hosted by Indexer01, I receive the expected results. However, when I search for data hosted by Indexer02, I get no results. I've restarted the Splunk services many times, I've removed and re-added the "failing" peer, to no avail. Logs are "clean" in that I'm not finding any glaring errors in both the "splunkd" and "remote_searches" log files.

What am I missing? Can a search head only utilize a single peer? What log should I be looking at?

Thank you in advance!
-Todd

Tags (2)
1 Solution

TAE2112
Explorer

Okay, the resolution to this was almost painful in its simplicity.

The fix:

  1. On the search head, click Settings > Distributed Management Console
  2. Note with frustration that only one Indexer is listed. Click "Setup" in the little green menu bar near the top.
  3. See that both remote instances (Indexers) are listed, but the one which is failing is not listed as "Configured". (I think it was offline or something.)
  4. On the Indexer which isn't working, under the "Actions" column, select Edit > Edit Server roles.
  5. Make no changes to the list of roles. (Seriously - no changes.)
  6. Click "Save"
  7. Wait a moment, and then search for data specific to the second Indexer - the one which wasn't working.
  8. Cheer when you get data back.

I'm not sure how I missed this in the documentation, but that's all it took. Hope this helps someone else.

-Todd

View solution in original post

anand_singh17
Path Finder

Cause of this error:

  • Cluster-bundle applied, but SH was busy, could not take up the update.

Resolution:
1. restart the splunkd service on SH.
2. Rejoin the SH to the cluster

0 Karma

TAE2112
Explorer

Okay, the resolution to this was almost painful in its simplicity.

The fix:

  1. On the search head, click Settings > Distributed Management Console
  2. Note with frustration that only one Indexer is listed. Click "Setup" in the little green menu bar near the top.
  3. See that both remote instances (Indexers) are listed, but the one which is failing is not listed as "Configured". (I think it was offline or something.)
  4. On the Indexer which isn't working, under the "Actions" column, select Edit > Edit Server roles.
  5. Make no changes to the list of roles. (Seriously - no changes.)
  6. Click "Save"
  7. Wait a moment, and then search for data specific to the second Indexer - the one which wasn't working.
  8. Cheer when you get data back.

I'm not sure how I missed this in the documentation, but that's all it took. Hope this helps someone else.

-Todd

knulps
Engager

Thanks for this solution. it worked for me. I went ahead and made my Monitoring Console as standalone.  that fixed the issue. 

0 Karma

Kieffer87
Communicator

Not sure I would have thought to do this since I don't use this search head as my DMC.

0 Karma

divyamudundi
Path Finder

This fixed the issue when our search head peer has problem communicating to one of the remote instance.

0 Karma

RecoMark0
Path Finder

This has worked for me multiple times, thank you. I seem to have an issue every time I restart my indexers, where no data comes in to the search head. I just "fake edit" the two indexers in the server roles column, and apply changes. After that everything works!

0 Karma

jdaves
Path Finder

Thank you. I've added "Spin through DMC and save configuration any time architecture is modified" to our checklist... ugh.

0 Karma

bandit
Motivator

Splunk please fix this bug!!! I would have never figured this out without the above answer. I ran into this issue by removing a search peer then added it back and no longer got search results back from the search peer. The workaround above worked for me. Not sure why the DMC interacts with distsearch.conf config.

0 Karma

phoffman_splunk
Splunk Employee
Splunk Employee

Define your index in your search from search head index=

Also, verify that users roll has permissions to search that index (in Settings >> Access controls >> Roles)

You can verify that the distributed search is quering both peers by checking the job inspector (after search, click on the "job dropdown")

0 Karma

TAE2112
Explorer

Thanks for the response!

Bad news: Unfortunately, it didn't have anything to do with roles

I thought you may have hit upon the piece I was missing - the "role permissions" suggestion - but it didn't change anything, as the request is still not returning data. Further, I've been using the working indexer as a point of reference (set up by someone else), and this server doesn't have anything special set up with regards to permissions and indexes.

The job inspector feature, however, does confirm the fact that only one of the two indexers (the one which is working) is being queried for information based on my search criteria. The other indexer is mysteriously absent, even though it shows as being "up" in the list of search peers.

I've wrangled some time with the individual who set up the original (working) indexer as a peer, so I hope I'll get an answer shortly.

0 Karma

Raghav2384
Motivator

Is Indexer 2 and Universal Forwarder communicating? By your description, looks like indexer 1 is all good.

Check your outputs.conf and inputs.conf on the UF again. Can you launch splunkweb on the Indexer2 and see if it shows any events in the data summary wiz?

Thanks,
Raghav

0 Karma

TAE2112
Explorer

Yep - all communication is successful. UF's on the servers are all reporting to their respective indexer (the indexer specific to that datacenter). The Search Head server and Indexers are both talking to one another. It's all very weird - if they can talk, why can't they exchange information?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...