How to create and calculate a response time graph?

[maniishpawar]

How do i calculate every 10 seconds, the average response time for the past 5 minutes and plot on a graph.

[DalJeanis]

Okay, if you are on splunk below 6.4, then streamstats won't work for you. here's an alternate route. Basically, we copy each record forward into the next twenty-nine 10-second intervals, kill the excess records that go out into the future, and then let timechart do all the work.

  your query to return fields _time and responseTime over a 15-minute interval 
 earliest=-16m@m 
  | bin _time span=10s
  | eval fan_time=mvrange(0,300,10)
  | mvexpand fan_time
  | eval _time=_time+fan_time
  | addinfo
  | bin info_max_time as maxtime span=10s 
  | where (_time >= info_min_time+300) and (_time < maxtime)
  | timechart span=10s avg(responseTime) as avgResponseTime

[lguinn2]

If you want to plot a moving average, perhaps you can use the trendline command. I also like @somesoni2's answer too.

 your query to return field 
| sort _time
| trendline sma30(responseTime) AS trend
| table _time responseTime trend

However this answer makes a different assumption about the data. This assumes that you collect responseTime information every 10 seconds. To get a 5-minute average at any point, you would need to average at the prior 30 responseTimes. This is what trendline does.

[somesoni2 · Accepted solution]

Assuming Splunk Cloud is using Splunk version 6.4 or above (check Help-> About), give this a try

your query to return field responseTime 
| bucket span=10s _time
| streamstats time_window=5m avg(responseTime) as "Average Reponse Time"
| dedup _time | table _time  "Average Reponse Time"

[DalJeanis]

For a smidge more accuracy... and ONLY a smidge...we'd probably do the bucket after the streamstats, and then run it into stats... or just ignore the buckets and run it directly into timechart, like so ...

 your query to return fields _time and responseTime over a 15-minute interval 
earliest=-16m@m 
 | sort 0 _time  
 | streamstats time_window=5m avg(responseTime) as avgResponseTime
 | addinfo
 | bin info_max_time as maxtime span=10s 
 | where (_time >= info_min_time+300) and (_time < maxtime)
 | timechart span=10s avg(avgResponseTime) as avgResponseTime

... It's probably not absolutely necessary to kill the most recent interval, which will never be 10s long, but we did it anyway in that addinfo code. You know, "belt, suspenders, safety pins and duct tape".

[maniishpawar]

sorry @somesoni2 , i'm here again. what is the equivalent of streamstats in splunk 6.3.3 ?
or is there a different way to write this in older splunk version

[maniishpawar]

I tried this query , its showing the results.
when I click on one timestamp (10s each) , should this be showing me the events of -5m from that timestamp.
As of now, when I clicked lets 4/20/2017 1:20:00 PM timestamp it does not show me the first event being 4/20/2017 1:15:00 PM and last being 4/20/2017 1:20:00 PM

[somesoni2]

My guess will be no, it won't show you events for 5 min window of the time clicked. It will show the events from time clicked + the timechart span which is 10 sec. For showing results for last 5 min you'll have to setup custom drilldown to take the clicked timestamp and update earliest and latest accordingly.

[woodcock]

You create a dashboard that refreshes every 10 seconds (actually, this is going to be WAY too much flashing, so I suggest every 30 seconds at least) and add a single panel to that dashoard that runs a single search and generates a single-value visualization. Then you create the search that drives that panel, something like this:

Your Base Search Here earlieast=-5m@m | stats avg(responseTime) AS AvgResponseTime

[woodcock]

Never mind; I missed the "and plot as a graph".

[somesoni2]

So at 10:00:00, show average of 09:55:00 to 10:00:00, at 10:00:10, show average of 09:55:10 to 10:00:10, at 10:00:20, show average of 09:55:20 to 10:00:20, and so on...?

[maniishpawar]

Yes correct

[somesoni2]

What version of Splunk are you using?

[maniishpawar]

we are using splunk cloud

[gokadroid]

let's say that the response time is being taken in a field called responseTime and the statement calculate every 10 seconds, the average response time for the past 5 minutes and plot on a graph means what is the average response time calculated for 10 sec spans and when watched over last 5 minutes, here is what you can try:

your query to return field responseTime earliest=-5m
| timechart span=10s avg(responseTime)

[maniishpawar]

little bit confusing, but to me the answer seems providing average on 10 sec window,
but the avg is required for previous 5 mins. please correct me if I am wrong.

so all in all for 1 hour we will 60*6 =360 samples( each at 10s interval) , each showing me the average of past 5 mins from the collected _timestamp.