Splunk Search

Percentile Implementation

sohrab
Explorer

Hi

I am wondering what percentile implementation does Splunk use (used by stats, etc.). It does not always return the same results as Excel's or what I calculate manually (may be interpolated).

Is it the function from scipy.stats? Or it is a custom function? Is it possible to get the formula if it is custom?

Tags (2)
1 Solution

steveyz
Splunk Employee
Splunk Employee

If there are less than 1000 distinct values, the percentiles use the nearest rank algorithm (see http://en.wikipedia.org/wiki/Percentile#Nearest_rank). Excel uses the NIST interpolated algorithm, which basically means you can get a value for a percentile that does not exist in the actual data, which is not possible for the nearest rank approach. You can ask splunk to use the excel method instead via a limits.conf setting [stats] perc_method=interpolated (vs 'nearest-rank'). See the limits.conf.spec entry for more detailed info.

If there are more than 1000 distinct values for the field, the percentiles are approximated using a custom radix-tree digest based algorithm that is much faster and uses much less (a constant amount) memory than an exact computation (which uses memory in linear relation to the number of distinct values). By default this approproach limits the approximation error to < 1% of rank error. That means if you ask for e.g. 95th percentile, the number you get back is between the 94th and 96th percentile.

You always get the exact percentiles even for more than 1000 distinct values by using 'exactperc' instead of 'perc'

View solution in original post

Ellen
Splunk Employee
Splunk Employee

For additional reference, in 4.3.2 you can find further details in the following files.

$SPLUNK_HOME/etc/system/default/searchbnf.conf

[stats-perc]
syntax = (perc|p|exactperc|upperperc)
simplesyntax = perc
description = The n-th percentile value of this field. perc, p, and upperperc give approximate values for the integer percentile requested. The approximation algorithm we use provides a strict bound of the actual value at for any percentile. perc and p return a single number that represents the lower end of that range while upperperc gives the approximate upper bound. exactperc provides the exact value, but will be very expensive for high cardinality fields.

$SPLUNK_HOME/etc/system/README/limits.conf.spec

perc_method = nearest-rank|interpolated
* Which method to use for computing percentiles (and medians=50 percentile).
* nearest-rank picks the number with 0-based rank R = floor((percentile/100)count)
* interpolated means given F = (percentile/100)
(count-1), pick ranks R1 = floor(F) and R2 = ceiling(F). Answer = (R2 * (F - R1)) + (R1 * (1 - (F - R1)))
* See wikipedia percentile entries on nearest rank and "alternative methods"
* Defaults to interpolated

0 Karma

steveyz
Splunk Employee
Splunk Employee

If there are less than 1000 distinct values, the percentiles use the nearest rank algorithm (see http://en.wikipedia.org/wiki/Percentile#Nearest_rank). Excel uses the NIST interpolated algorithm, which basically means you can get a value for a percentile that does not exist in the actual data, which is not possible for the nearest rank approach. You can ask splunk to use the excel method instead via a limits.conf setting [stats] perc_method=interpolated (vs 'nearest-rank'). See the limits.conf.spec entry for more detailed info.

If there are more than 1000 distinct values for the field, the percentiles are approximated using a custom radix-tree digest based algorithm that is much faster and uses much less (a constant amount) memory than an exact computation (which uses memory in linear relation to the number of distinct values). By default this approproach limits the approximation error to < 1% of rank error. That means if you ask for e.g. 95th percentile, the number you get back is between the 94th and 96th percentile.

You always get the exact percentiles even for more than 1000 distinct values by using 'exactperc' instead of 'perc'

sohrab
Explorer

Thanks for the speedy response.

0 Karma

sundarrajan
Path Finder

@steveyz,
Dear Steve, is it possible for us to get a sneak peak into rdigist algorithm or any "custom built radix tree digist algorithm" for knowledge purpose. In 6.4 we could see by-default splunk takes "closerank" algorithm over "interpolated".

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...