search is always "parsing...." after upgrading to 6.5 from 6.3.2 - SPL-129476 This is actually caused by stale files remaining in cache after the upgrade. If you haven't all ready done so, try clearing browser cache and retry the search. This has helped quite a few customers reporting this issue. ... View more

Once the customer got in touch with Microsoft, he was able to narrow it down to the "Dfsr" performance counter using exctrlst.exe (see: http://blogs.technet.com/b/askperf/archive/2010/03/05/two-minute-drill-disabled-performance-counters-and-exctrlst-exe.aspx ). Once he disabled that counter all of their normal counters returned, and are now being forwarded correctly. ... View more

You can change the location of Splunk's internal log set by editing the #SPLUNK_HOME/etc/log.cfg file. For instance: I want to change the location of the splunkd.log and rename it. Look for the following entry: appender.A1.fileName=${SPLUNK_HOME}/var/log/splunk/splunkd.log You can change the location to something like: /var/log/splunk/splunk_indexer.log Disclaimer Splunk Support does not advise to do this as it can have adverse effects. One of which is likely to be the disruption of indexing internal logs. ... View more

This number can vary depending on the exact numbers of clients and app changes, as well as how many of the clients are hitting the deployment server at the same time. A rule of thumb being no more than 2000 clients hitting within a minute – after that – we have to start increasing the phone home interval. Generally somewhere in the region of 600 - 1200 seconds should be a good place to start. Deployments of 3000 potential clients should have a phone home interval of 5 - 8 mins. We do have customers with 15,000 clients who adjust 'phoneHomeIntervalInSecs' and 'handshakeRetryIntervalInSecs' parameters to be successful. You can also specify a different check-in interval on a per client, or per class basis by tuning the 'phoneHomeIntervalInSecs' and 'handshakeRetryIntervalInSecs' parameters in deploymentclient.conf - http://docs.splunk.com/Documentation/Splunk/latest/Admin/Deploymentclientconf ... View more

We checked the following as possible causes for this issue: outputs.conf for multiple entries using same port more than one instance of Splunk running Firewall issues Event Logs show: Audit Failure 6/10/2013 10:08:37 AM Microsoft Windows security auditing. 5157 Filtering Platform Connection Audit Success 6/10/2013 10:08:37 AM Microsoft Windows security auditing. 5156 Filtering Platform Connection Audit Failure 6/10/2013 10:08:37 AM Microsoft Windows security auditing. 5157 Filtering Platform Connection Audit Success 6/10/2013 10:08:37 AM Microsoft Windows security auditing. 5156 Filtering Platform Connection The issues was fixed by this Microsoft KB article: http://support.microsoft.com/kb/2654852 ... View more

This isn't technically a Splunk issue, but here's how we resolved it: We were seeing the 500 errors, and also noticed this issue when using a Windows configuration tool that uses a phone-home style of connection. We saw the following error from that software: Could not send report: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full. - connect(2) After searching Microsoft KB articles for the error, we found the following article that resolved the issue: Title: When you try to connect from TCP ports greater than 5000 you receive the error 'WSAENOBUFS (10055)' Symptom: If you try to set up TCP connections from ports that are greater than 5000, the local computer responds with the following WSAENOBUFS (10055) error message: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full. http://support.microsoft.com/kb/Q196271 Once I applied the change from the KB article, connections started flowing again to the operations system. The machine was able to authenticate to our windows domain, Splunkd started making connections to other machines to index, and we could connect to outside servers. ... View more