I have created a saved search that runs every minute. I have opted to run a perl script as the alert option. Splunk passes in the search id to the script as one of it's parameters. Whenever I try to retrieve the events via REST, I get a blank page.
Here's what I run to get the events:
https://server:8089/services/search/jobs/my_search_id/events?output_mode=raw
I AM able to get the RESULTS though:
https://server:8089/services/search/jobs/my_search_id/results
Splunk tends to kick off searches in the most efficient way possible. One way it speeds things up is that any fields that aren't referenced in the search wont be extracted or summarized. But another way is that unless someone sets the 'status_buckets' argument (on the dispatch POST) to something greater than 0, then it will preserve only the final results
-- it wont preserve the raw events nor any bucketed summary information by time.
Note of course, if the search you are running does NOT have any reporting commands on it, the results and the events are the same -- ie you'd get the exact same results from the results
endpoint as you would from the events
endpoint.
But anyway when you're running scheduled reports, just set
dispatch.buckets = 1
and that will basically tell Splunk to keep the events around, accessible through the events
endpoint, and incidentally also the field summaries around, accessible through the summary
endpoint.
You can see a little more background in this other answer:
http://splunk-base.splunk.com/answers/4028/job-summary-endpoint-fatal-error
And the UI Examples app has some pages talking about how status_buckets
and it's cousin required_field_list
are set by the views, and how this can effect performance. For more reading there pull down the "ui examples" app from Splunkbase and under "Advanced XML" you'll see several views with examples and docs, called "Affecting Search Performance".
And they are mentioned briefly in the REST docs although really I wish it would explain them a little more:
http://www.splunk.com/base/Documentation/4.2.1/Developer/RESTSearch#POST
What are the implications of this change to the performance of splunk?