The Read Operation Timed Out - how do i stop this occurring?!

[pj]

I am developing a dashboard that leverages the HiddenSavedSearch module to bring in saved search results set that contains about 250k events.

I am then using this search to populate a number of panels in the dashboard, where i am postprocessing the events to display summary tables, pie charts etc. I then allow the user to drill down to the results set.

The problem seems to be that being a big chunk of data to postprocess, it is taking possibly longer than Splunk will permit. As a result all my panels are displaying:

Results Error: 500 - Splunkd daemon is not responding: ('The read operation timed out',)

So, what would be the workaround here? I assume there might be some tweak i can make to extend the timeout period? On the odd occasion that the postprocessing completes within whatever the limit is, the dashboard renders fine. I cannot reduce the number of events as we have a lot of data to report on. Bringing the results back in this fashion using a hiddensavedsearch is a lot faster than actually searching for them directly.

We are using Splunk 4.3.2

Thanks!

[pj · Accepted solution]

Leverage summary indexing, report acceleration or data model acceleration. The new features of Splunk 6 should mean that there are other ways to render large amounts of data extremely quickly and in a manner that scales.

[jluste]

I'm using 5.0.8 and I have the same issues for certain post processed panels on a dashboard. The dashboard uses 1 saved (accelerated) search for like 15 panels. 1 or 2 of the panels depending on the time picker choice, get time outs. Is there a control for this?

[pj]

Not using pooled search heads - pooling has proven to be challenging. So we are just using regular SH and indexers. Correct use of summary indexing negates the need to even use hiddensavedsearch in this manner.

[richnavis]

I had lots of problem with this error when I used pooled searchheads.. once I went away from them.. they went away.. What is your architecture?

[pj]

Blast from the past - thanks for the comment reminding me about this. So I actually gave up on this as I could not find a workable solution. After all sorts, of other trial and error, we ended up using summary indexing and then running the dashboards from the summary index. The approach i originally used should have worked, but it doesnt (maybe a bug, who knows). If you keep the fields in your summary down (v. important for speed), you should be able to get a pretty fast render in the dashboard - usually 10secs or so.

[cpeteman]

Did anyone ever fix this problem?

[vmorel]

Same issue here, same version and also with a HiddenSavedSearch. No solution so far. It only happens when I change the time to more than 4 hours.