Splunk Search

Why are REST API searches returning intermittent results?

ltrand
Contributor

I'm having an weird situation where REST queries sometimes pull results and sometimes don't. We've even tried limiting the search to just return 10 results with | head 10 after the gather stage. I see the traffic in PCAP for all request attempts, but some requests just fail to return any results. For those that fail, I don't see any log entries for the search. For all that returned results, I see log entries for the search activity. The same search can fail and then succeed or vice versa.

What should I start investigating next? Has anyone seen an issue like this before?

Tags (2)
0 Karma

javiergn
SplunkTrust
SplunkTrust

Have you tried adding the following to your query?

splunk_server=local

For instance, who am I?

| rest /services/authentication/current-context splunk_server=local | table username

I remember getting weird results when running REST queries on the DMC server and forcing the queries to run against the local server fixed the problem

0 Karma

jplumsdaine22
Influencer

What does the _internal index say about those SIDS ?

0 Karma

ltrand
Contributor

That's what's killer, the ones that fail have no entries for the user in _internal at all. It's like it didn't even register the request as valid.

0 Karma

jplumsdaine22
Influencer

Whats in the HTTP response? EG: Do you get any response at all from Splunk or are your requests just timing out.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...