Splunk Search

Charting percentage of a total over time?

Jason
Motivator

I'm working with some HTTP access logs that have a status code in them. Most are successful messages, naturally. I would like to see a chart, over time, that plots the percentage of messages that are errors, to avoid daily cycles in amount of hits and see if a growing proportion of the messages are error messages.

There doesn't seem to be this "percentage of whole" function in stats / chart / timechart. What can I do?

I basically need the "percentage" column from top, over time.

1 Solution

Jason
Motivator

This is a search that seems to be the shortest possible way to do this. The idea is:

  • break out events into a graphable number of discrete time buckets
  • get a total number of events per time bucket
  • count up the number of each status by time bucket, keeping the totals intact
  • calculate the percentage for each status/time bucket combination
  • throw out any status you do not want to graph (for example, success messages)
  • timechart the percentage by status
... | bucket _time bins=100
| eventstats count as total by _time 
| stats count first(total) as total by _time, http_response 
| eval percent=(count/total)*100 
| search NOT http_response="2*" 
| timechart first(percent) by http_response

This will give you a graph of percentage numbers by each type of error over time. The Y axis will likely be from 0-1 (as in percent) for a properly functioning system.

View solution in original post

jiman7697
Explorer

I have perhaps a better solution for those who seek to get a percent success broken down by some other field over time.

This gives percent success over time by a field "url" in some http logs. Just configure the span in the bucket command to control the time split, and add "%H:%M:%S" to the time format if you need hours/minutes/seconds.

index=my_http sourcetype=http_logs http_status_code IN (2*, 3*, 5*)  
| bucket _time span=1d 
| eval success=case(match(http_status_code ,"2.*"), "1", match(http_status_code ,"3.*"), "1", match(http_status_code ,"5.*"), "0") 
| eventstats count as total, sum(success) as successes by url, _time
| eval pct=round((successes/total)*100,2)
| eval timestring=strftime(_time, "%m-%d-%y")
| chart first(pct) by url, timestring
0 Karma

lmachetman
Explorer

Perhaps this wasn't available in earlier versions - but with latest splunk you can change your stackmode to 100% stacked - here's what it generates in XML:

  <option name="charting.chart.stackMode">stacked100</option>
0 Karma

Jason
Motivator

This is a search that seems to be the shortest possible way to do this. The idea is:

  • break out events into a graphable number of discrete time buckets
  • get a total number of events per time bucket
  • count up the number of each status by time bucket, keeping the totals intact
  • calculate the percentage for each status/time bucket combination
  • throw out any status you do not want to graph (for example, success messages)
  • timechart the percentage by status
... | bucket _time bins=100
| eventstats count as total by _time 
| stats count first(total) as total by _time, http_response 
| eval percent=(count/total)*100 
| search NOT http_response="2*" 
| timechart first(percent) by http_response

This will give you a graph of percentage numbers by each type of error over time. The Y axis will likely be from 0-1 (as in percent) for a properly functioning system.

helge
Builder

Awesome!
Make sure to replace the HTML emphasis (<em> and </em>) with asterisks (*).

0 Karma

Glenn
Builder

You're a lifesaver!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...