I'm having an weird situation where REST queries sometimes pull results and sometimes don't. We've even tried limiting the search to just return 10 results with | head 10
after the gather stage. I see the traffic in PCAP for all request attempts, but some requests just fail to return any results. For those that fail, I don't see any log entries for the search. For all that returned results, I see log entries for the search activity. The same search can fail and then succeed or vice versa.
What should I start investigating next? Has anyone seen an issue like this before?
Have you tried adding the following to your query?
splunk_server=local
For instance, who am I?
| rest /services/authentication/current-context splunk_server=local | table username
I remember getting weird results when running REST queries on the DMC server and forcing the queries to run against the local server fixed the problem
What does the _internal index say about those SIDS ?
That's what's killer, the ones that fail have no entries for the user in _internal at all. It's like it didn't even register the request as valid.
Whats in the HTTP response? EG: Do you get any response at all from Splunk or are your requests just timing out.