On event actions under show source my users are getting the following error:
Streamed search execute failed because: Error in 'surrounding': Too many events (> 10000) in a single second.
No search results for surrounding search where targetId=239:112912518
I have two search heads and this error only occurs on one search head and I can't find anything different in the two. I only have one indexer at this point.
I ran into the same issue today. I was able to find a parameter in limits.conf
When I bumped it up, the problem was solved. I don't know how it will affect performance, so I'll probably take it out. I'm pretty sure it only occurred because we have temporarily put some logs in debug.
I created /opt/splunk/etc/system/local/limits.conf with this.
[show_source]
#maximum events retriveable by show source
max_count = 50000
I have some users getting this same error, but others who don't for the same event. I set max_count = 50000 in limits.conf and restarted Splunk, but the error message still happens, and still says 10000. I looked through the logs to see if I could verify that the setting took, but haven't been able to find it yet.
Any suggestions for how I can verify the higher limit? Are there perhaps some capabilities that would make show source behave differently for users in different roles?
I have found my answer in the following:
https://answers.splunk.com/answers/351000/streamed-search-execute-failed-because-error-in-su.html
The max_count setting in [show_source] must be set on the Search Peers, as the Search Head doesn't push that value when requesting the search.