no events returning for any search jobs (api call)

[hiddenkirby]

i am not recieving any xml for /services/search/jobs/<sid>/events

... but i get xml for /services/search/jobs/<sid>/results

I'm not sure i entirely understand the difference between the two... but this endpoint doesn't seem to work for ANY search.

halp?

[Lowell · Accepted solution]

Yeah, I confused by this too the first time I tried to search via the REST API.

I think it's easier to understand based on the new UI in Splunk 4 than it was in 3.x.

If you run a search from the UI, there are three different tabs you can switch between. The first two show you your "events" ("Event list", and "Event table" modes), and the final tab is the "Results table".

So basically your "events" are untransformed events. This includes the _time and _raw fields. Generally this is anything that happens before a stats or chart search command. The "results" are only available if you do some sort of transformation and would only include the fields explicitly listed in your stats command, for example.

To make matters worse, it seems that sometime the terms "events" and "results" are used interchangeably.

If you are running 4.1, you can use the 'Job Inspector' to see the different between these 2. Look for the entry eventSearch vs the reportSearch. There is also field called search that shows the combination of both of these, and I think will always match your actual search string.

Another gotcha here is that if you are doing a transformation on your events and running the search from the UI than you can see both the events and the results. I don't think it works this way by default for REST searches. Try setting status_buckets=300 when you submit your search job. (This is similar to the way a saved search runs vs running a search interactively.... It's more efficient for splunk NOT to keep both event and results when it doesn't have to. There can be a big savings by simply not extracting all the fields when splunk knows their aren't going to be used.)

[Lowell]

It may also help to poke around in the $SPLUNK_HOME/var/run/splunk/dispatch/<jobid> folders. There you should can see what splunk is actually storing for a given search job. For example, you'll see the results in results.csv.gz and your events in the events folder along with other files that correspond to the various endpoints. (Of course just browsing the endpoints with a browser can be helpful too.)

[hiddenkirby]

Thank you. This helps a lot. It still raises my eyebrow... but it points me in a good direction. Thanks Lowell.