We came across the following issue in production: after starting an export job and receiving a few hundred thousands of events back, the export job stopped returning any more events, yet the connection was still open, and we did not get any exceptions (the API call was blocked waiting on more events). Basically, the program was hanging waiting to read more events from the ResultsReader (we're using the Java API).
Unfortunately this case is quite difficult to reproduce. However, we still want to avoid something like this happening in the future. Is there a way to forcefully timeout the connection in case no more events were being received via the ResultsReader? There is a "setAutoCancel" call for the JobExportArgs API (http://docs.splunk.com/DocumentationStatic/JavaSDK/1.4.0/index.html?com/splunk/JobExportArgs.html), though it is somewhat confusing because the docs say that an export search does not create a new job on the Splunk server:
http://dev.splunk.com/view/java-sdk/SP-CAAAEHQ
Export: An export search runs immediately, does not create a job for the search, and starts streaming results immediately. This search is useful for exporting large amounts of data from Splunk Enterprise.
Right now, one course of action is to implement a timeout mechanism on our side, and forcefully restart the search query. Is there a better way?
... View more