Monitoring Splunk

Why is a REST call so slow

sloshburch
Splunk Employee
Splunk Employee

Just saw an environment where it was taking 15 seconds to run | rest /services/search/jobs splunk_server=local while in another environment it took 0.121 seconds (about a tenth of a second!). Why would a simple rest call, such as this, take so dang long?

Anyone have insights?

The Search Job Property "Custom" was:

{
"dispatch.earliest_time": "‐15m",
    "dispatch.latest_time": "now",
    "dispatch.sample_ratio": "1",
    "display.general.type": "statistics",
    "display.page.search.mode": "fast",
    "display.page.search.tab": "statistics",
    "search": "| rest /services/search/jobs splunk_server=local"
}

alt text

0 Karma
1 Solution

sowings
Splunk Employee
Splunk Employee

The jobs endpoint has to iterate over all of the entries in the dispatch directory. More jobs = more time. It could also come down to storage performance on the SH itself.

What were the respective number of results found between the two environments?

View solution in original post

sowings
Splunk Employee
Splunk Employee

The jobs endpoint has to iterate over all of the entries in the dispatch directory. More jobs = more time. It could also come down to storage performance on the SH itself.

What were the respective number of results found between the two environments?

jnguyen413
New Member

Working with SloshBurch on this, and the number of events that we receive are 2817 events. Is that a lot?

0 Karma

sowings
Splunk Employee
Splunk Employee

Considering that the default limit where you start to get warnings from Splunk on a high number of dispatch directories is 3000, yeah, it's a lot. Consider that to get information about the jobs you have to go read through some metadata in each of those dispatch dirs, so it's now about speed of IO on that SH.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...