Why do I see a count difference running a saved search via Splunk REST API call and running the search manually?

[senthilponnuswa]

When I run a saved search via Splunk REST API call, I get a count which is entirely different when iI run the same search manually on the Splunk server. Will there be any difference in getting the count via API or running manually?

[purnavenkatesh]

HI,

I am facing the same issue. How did you solve this issue?

[rfujara_splunk]

Assuming you don't specify an absolute time range but using relative time range settings like earliest=-10m and latest=now.

Then you run two searches (regardless if REST API first and then Splunk UI or the other way round), there is no guaranty that the searches are executed at the same time within the Splunk server. Because of that, the relative time ranges result into different absolute time ranges. If new events come in, within the delta time of the two absolute time ranges, that may cause the different results.

[senthilponnuswa]

The scenario is like, when i run the API and SPlunk UI query together, the API is taking more time when compared to UI. But i get the samples output as same. But when i scheduled the API to run for every 1 hour and when i get the error count in terms for 5k. At the end of the day, when i run the splunk UI query for that particular time where it showed 5k, but for the UI it shows as less than 100. Due to this, it causes fault alerts generation to monitoring team.

Eg:
Splunk API call run for every 1 hour. Each hour the count is arround 4k.

End of the day, i run a query from UI for time slice of every 1 hour.. Now i get the count as less than 100 for each hour. Where as API output has logged as 4k for every hour.

[s2_splunk]

Are you sure you are using the same time range on both queries?
The same exact query with the same exact time range should not return different results depending on invocation method.

[tikoonikhil]

I am also facing the same issue. When i do a query through the API for some particular time interval, it gives me some count. When i do the same query for the same time interval through the search via UI, the count that i get is different.
Also, does difference in timezone have any effect on the search results obtained from querying through API and querying through UI?
Thanks in advance.

[nathanielduren]

We are having the same issue. Run via the API and the results are 160. Run it manually and we get thousands of results. Any luck on solving this issue?

[senthilponnuswa]

When i mention earlier and latest time range and hard code the query in search , i'm able to get the same data which i get while i run manually. But when i run the query with span of last 60 minutes, it gives weird output. Eg: the failure count via manual query is 40 and via API i get 4751. Since because of this, it creates a fault graph and generates fault alarm.