How to calculate an average of P98 of last 5 different search request together

[neeldesai1992]

My task is to calculate the average of P98 of last 5 requests and compare it with the latest request's response time. I am new to splunk so how can I calculate the average of P98 of last 5 runs(request's response time) & compare it to the current(latest) request's response P98?

[DalJeanis]

There is a disconnect in what you are asking.

P98 is the 98th percentile. It is a calculation, similar to an average. Basically, if you have a hundred people and measure their heights, the P98 is the guy standing next to the end.

"The average of the P98" is not meaningful by itself, if you are dealing with only 5 readings of one measurement. You could take the average of the P98() of the response times for a number of DIFFERENT hosts, and that would be a meaningful measurement regarding how all the hosts as a group were acting.

We're going to assume that you are being asked to calculate the 98th percentile of the last 5 responses.

Let's also assume that you know about how often these transactions occur, so you can start your query far enough back to pick those up. We'll discuss that more later.

This gets you any records that are higher than the P98().

 Your search that gets _time and responsetime, at least 6 of them
| streamstats current=f window=5 p98(responsetime) as P98resp 
| where responsetime>= P98resp

---

GRABBING EXTRA FOR THE CALCULATION

Now you have to put that in context of how often you need to run. You need to start far enough back that you know there will be 5 records or more.

So, for example, if there are about eleven records a minute then you need an average of about 27 seconds worth of prior data. We could calculate an exact required duration based on assuming a Poisson distribution -- but that would just be showing off -- so instead we'll double our number and round it up to a minute.

Let's assume you are running the query every five minutes to pick up any slow responses during that period. Therefore, we would need to have the query starting 6 minutes ago and ending at the current minute and, after calculating the P98(), we would throw away the first minute's worth of traffic so it wasn't reported a second time.

earliest=-6m@m latest=@m
Your search that gets _time and responsetime
| streamstats current=f window=5 p98(responsetime) as P98resp 
| addinfo
| where (responsetime>= P98resp) 
     AND (_time>=info_min_time + 60)

---

ADDITIONAL FIELDS

If there were multiple hosts for which you were doing this, then you would add the host field in a couple places...

 earliest=-6m@m latest=@m
 Your search that gets _time, host and responsetime
| streamstats current=f window=5 p98(responsetime) as P98resp by host
| addinfo
| where (responsetime>= P98resp) 
     AND (_time>=info_min_time + 60)

[neeldesai1992]

Thanks DalJeanins but this is little bit confusing to my main question. so let me describe this more in details so we have a jenkin job that upload a log file on splunk. So let's there are 6 builds of which log file has been uploaded on splunk. Let's take it as build#1,build#2,build#3,build#4,build#5,build#6. Now take build#6 as a latest build. Now I want to calculate the P98() of the latest build(#6) and compare it to average of previous builds P98(). Now can you tell me which splunk search query would help me to do so? So far I have following search query which I think so returns response time of build #6.

index=cp source=FT buildNumber=6 type=REQUEST | rename wholeduration as duration1