In savesearches.conf there is a section that specifies what should occur if a search peer disappears or is added.
restart_on_searchpeer_add = 1 | 0
* Specifies whether to restart a real-time search managed by the scheduler when a search peer
becomes available for this saved search.
* NOTE: The peer can be a newly added peer or a peer that has been down and has become available.
* Defaults to 1. - http://docs.splunk.com/Documentation/Splunk/6.1/Admin/Savedsearchesconf
This doesn't appear to apply to searches run from the realtime app however (which are actually triggered from inputs.conf).
Apart from running some search that detects this and then doing a manual restart/curl api reload etc is there anyway to fix this?
Not easily. To be honest, I don't think that this setting existed when I created the initial app. There is some setting posted to the search/jobs API that enables that functionality, but I can't even figure out what that setting is:
http://docs.splunk.com/Documentation/Splunk/6.1.1/RESTAPI/RESTsearch#POST_search.2Fjobs
Hahahaha you might be right regarding the restart option. I'll pass along your feedback to product management.
Thanks. We're trying to build in some robustness around this app as we're putting it into a production environment feeding data to arc sight.
fyi, the restart option was added in splunk v4.2 as far as I can tell.
That setting defaults to 1 anyhow so it shouldn't need to be set.