Splunk search head issues running on a VM (memory leak?)

[bruceclarke]

All,

I have a search head running on a VM that reads from two search indexers (also on VMs). I've been having issues with the search head's virtual machine.

About once every week or two, we can no longer log onto the head's VM and the Splunk front end won't allow logins either (the front end gives us a ResponseNotReady error). When trying to log onto the VM, we get an error saying:

"Remote Desktop cannot verify the identity of the remote computer because there is a time or date difference between your computer and the remote computer. Make sure your computer's clock is set to the correct time, and then try connecting again. If the problem occurs again, contact your network administrator or the owner of the remote computer."

After we reboot the machine, everything goes back to normal.

We initially thought that there was an issue with the time on our VM, but further inspection showed that the time was right and the machine was also failing to do domain authentication in general. Our best guess now is that there is a memory leak issue with the Splunk service (though this is a little odd, since the indexers run just fine).

Has anyone seen this issue with search heads on VMs before? Like I said, we don't see the issue with the indexers, which are also on VMs. Are there any limits I can set in limits.conf to prevent Splunk from using too much memory?

Thanks!

[Jason · Accepted solution]

This error is likely due to Splunk being unable to talk to itself. It's a known issue with Windows, see:

http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/Workaroundfornetworkaccessibilityiss...

This answer has the most detailed information, in terms of users' experience.

http://answers.splunk.com/answers/68368/splunk-web-throws-responsenotready-error

[bruceclarke]

Hey @dart, if you want to convert your comment to an answer, you definitely solved it for me! Thanks!

[bruceclarke]

watsm10 - I was able to reduce the occurrences of this issue by throttling the number of concurrent searches in the limits.conf file. That said, I still have seen the issue occasionally (maybe once a month). It looks like dart has the solution I've been looking for.

dart - Thanks for the link. I'll try it out and see if that's the issue. I see the error in the splunkd.log file, so I'm pretty confident this is the issue.

[dart]

It's possible you are seeing this windows issue - http://docs.splunk.com/Documentation/Splunk/5.0.2/ReleaseNotes/Workaroundfornetworkaccessibilityissu...

[watsm10]

Hi Bruce,

I'm currently having the same issues. I'm not sure what the issue is with our VM's either. Did you get anywhere with yours?