How to troubleshoot why I received an alert saying "indexer is not reachable"?

[charlesguo_2]

I am pretty new to Splunk. Guess what, the consultant has left and I was supposed to take care of Splunk. I got an alert saying "indexer is not reachable". I can ssh to the server, the CPU and disk are OK.
What else can I check?

[renjith_nair · Accepted solution]

It can be of different reasons but do the basic checks

• Check if splunk process is running on indexer. Switch to your splunk user on indexer and change the directory to $SPLUK_HOME/bin where SPLUNK_HOME is the directory where your splunk installed and run ./splunk status.

• Check if the indexer is accessible from search head on port 8089 (or the management port you used)

• Check if the firewall is open between search head and indexer if they are not on the same subnet.

[charlesguo_2]

It says it is running ok:

./splunk status
splunkd is running (PID: 3001).
splunk helpers are running (PIDs: 3002 3018 3089 3188 3294 17563 17713).

I also test the connection and it is OK. I also didn't see obvious errors from log files.

[renjith_nair]

run a search | eventcount summarize=false index=* index=_*|table server|dedup server and see if all of your indexers are listed under server. It might be a momentary n/w issue. Are you still getting the error?

[charlesguo_2]

A server reboot solved the issue, still don't understand why.
Thanks for the help!