Splunk Search

Indexer not searchable by Search head

wweiland
Contributor

I'm having a problem where I have 5 indexers and 1 search head. All 5 show up in the search peers under distributed search. I've verified through the metrics.log that the indexer is receiving data. When I perform a search however, I only see events from 4 of the indexers. I performed "index=* | stats count by splunk_server" and again only 4 indexers + the search head showed up. Has anyone seen this issue before? Thank you in advance for any help or direction that can be provided.

0 Karma
1 Solution

wweiland
Contributor

Well, I feel like a goof. It looks like it is a heavy forwarder (has a outputs.conf). I don't understand why it looks like a search peer and showing up in SoS, but you learn something everyday. Thanks to everyone who took the time to respond.

View solution in original post

0 Karma

wweiland
Contributor

Well, I feel like a goof. It looks like it is a heavy forwarder (has a outputs.conf). I don't understand why it looks like a search peer and showing up in SoS, but you learn something everyday. Thanks to everyone who took the time to respond.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

If you run this search, how many peers return count?

index=_internal earliest=-5m@m | stats count by splunk_server

This should give responses from all your indexers, and if you have your SH / Component boxes configured to forward their internal logs, those also.

If you're missing an indexer, try running the same search from the cluster master. If that doesnt return results, i would try rejoining the indexer to the cluster.

0 Karma

Jeremiah
Motivator

Are there any license errors recorded for this indexer? Check your license server console.

0 Karma

pradeepkumarg
Influencer

couple of questions..
Is the indexer in question up and running?
Did it ever reported back the data?
If not, is it actually getting the data from your forwarders? Check the outputs.conf if the indexer name/ip is mentioned correctly

0 Karma

wweiland
Contributor

It is up and running. I'm not sure if it has reported in the past as I'm new to this system. It is receiving 20% of the data. This was verified through the metrics.log and SoS.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...