In my system I have a searchhead and an indexer, both on a same switch. From time to time I am experiencing this error when running searches on the searchhead:
Failed to create result provider for remote peer 'indexer' at uri...
some error messages from the splunkd.log which might be related :
05-02-2012 13:22:10.508 -0400 WARN NetUtils - select_for timeout hit waiting for read
05-02-2012 13:22:10.508 -0400 WARN NetUtils - Bad select for loop rv = -2
05-02-2012 13:22:20.745 -0400 ERROR HTTPClient - Should have gotten at least 3 tokens in status line, while getting response code. Only got 0.
05-02-2012 13:26:46.572 -0400 WARN NetUtils - select_for timeout hit waiting for read
05-02-2012 13:26:46.572 -0400 WARN NetUtils - Bad select for loop rv = -2
05-02-2012 13:26:46.572 -0400 ERROR LMTracker - failed to send rows, reason='problem with message to master=https://192.168.16.100:8089, reason='Unable to connect to remo
te peer: https://192.168.16.100:8089 rc=2''
05-02-2012 13:28:47.500 -0400 WARN NetUtils
This system was working perfectly for more than a year and few days ago I started to experience this problem without any changes to the configs.
What could be the cause?
i know this is an old post, but if you are still listening, I would check this:
https://answers.splunk.com/answers/217/error-httpclient-should-have-gotten-at-least-3-tokens-in-stat...
It would be worth looking at:
-Is your splunk instance oversubscribed.
https://splunkbase.splunk.com/app/748/
-Are you having network issues.
I've got the same log errors appearing on my cluster master 6.0.3