Real-time searches keep running and consuming system resources

[helge]

This might be a bug in Splunk 6.0.1 (on Windows). I am building a web framework app. Each dashboard has a timerange view on it that affects the entire dashboard. Whenever I select a real-time window (e.g. "5 minute window") the searches on the dashboard get switched to real-time mode. This consumes a significant amount of CPU performance.

The problem is that neither stopping the individual searches through their respective searchcontrols (which I have put on the page) nor opening another dashboard stops the real-time searches. Basically you open three dashboards and you have a dozen real-time searches running which consume 100% CPU. The only way to stop those searches is through Activity -> Jobs in the upper-right corner.

Any workarounds for this? Can anyone confirm they see this, too?

[lmyrefelt]

you could try the New setting in web.conf;

splunkdConnectionTimeout =
* Number of seconds to wait before timing out when communicating with splunkd
* Must be at least 30
* Values smaller than 30 will be ignored, resulting in the use of the default value
* Defaults to 30

[helge]

This happens in my test environment with only a single server.

[lmyrefelt]

Contacting the Support could be an idea as well 🙂

[lmyrefelt]

I would also check disk i/o

Firewall rules ? maybe they are "cutting" the communication ?

It looked like in Our case that the searches were running "fine" however we never gave any visible result back due to Heavy i/o and or problem With time between Storage and Our search-heads. Eventually splunkd time-out in Our case.

Syncing the different hosts seemed to have helped and we also established a "job-server", doing all the schedule-searches for us.

I will check to see if i have written some other experiences as well .. this is just from the top of my head.

[lmyrefelt]

I dont have a solution for you, however i have experienced this myself. However we were at that time running 5.0.1 5.0.2 or perhaps 5.0.3 .

I would check so that all the servers are syncing their clocks against a Central ntp server. And that they are insync. The same goes for Your Storage if you use a nfs, cifs share.

I would also check the different search qoutas for the role running the search.

[yong_ly]

what happens when you select a time window that's NOT real-time? Do the searches re-run as normal searches?

If you close the dashboard then the searches should stop running. If you don't want real-time searches to be running, you can disable them from the timerange picker or limit it in the configuration files so not too many can be run at the same time.

[helge]

As I mentioned the real-time searches never stop running unless I manually stop them. Closing the dashboard definitely does not stop them.
When I select a time window that is not real-time I get additional searches.