Splunk Search

Smart-mode search emits only one row of stats but gets auto-finalized for disk usage

jspears
Communicator

I have a user whose monthly report search is being auto-finalized due to disk usage. I've ensured there are no other jobs hanging around, and it's being run in Smart mode. Here's what the search looks like (base search modified for public posting):

index=redacted
| stats latest(time_taken) AS time_taken latest(responseTime) AS responseTime by CorrelationId 
| eval responseTime=(time_taken * 1000) 
| stats count, avg(responseTime) as AvgTime, p99(responseTime) as p99, p95(responseTime) as p95, p90(responseTime) as p90, p50(responseTime) as p50

This is over tens of millions of events, but it produces only one row of output, as expected. My only theory so far is that the first stats output gets saved to dispatch along with the final output, but that seems... wrong.

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Your initial stats triggers a massive on-disk search, exhausting the user's dispatch space. On slack you mentioned you have a tiny duplication issue which made you add that first stats - it's probably not really relevant to your overall results. Try this:

index=redacted
| stats dc(CorrelationId) as count, avg(time_taken) as AvgTime, p99(time_taken) as p99, p95(time_taken) as p95, p90(time_taken) as p90, p50(time_taken) as p50
| foreach AvgTime p* [eval <<FIELD>> = <<FIELD>> * 1000]

Note that I've moved the eval to after the stats, then you don't have to do millions of evals for each single event.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

Your initial stats triggers a massive on-disk search, exhausting the user's dispatch space. On slack you mentioned you have a tiny duplication issue which made you add that first stats - it's probably not really relevant to your overall results. Try this:

index=redacted
| stats dc(CorrelationId) as count, avg(time_taken) as AvgTime, p99(time_taken) as p99, p95(time_taken) as p95, p90(time_taken) as p90, p50(time_taken) as p50
| foreach AvgTime p* [eval <<FIELD>> = <<FIELD>> * 1000]

Note that I've moved the eval to after the stats, then you don't have to do millions of evals for each single event.

sloshburch
Splunk Employee
Splunk Employee

Also, it might help to toss in the sourcetype. If there are many sourceytpes in the index then filtering on that in the base search could reduce the payload returned.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...