Splunk Search

Plot up or down state over time?

Jason
Motivator

I have a dataset I just created using transaction that shows when a particular service is down by pulling in the "service down" message and the "service up" message.

How do I plot when the service is up vs down on a graph over time?

Similar to the question here.

Jason
Motivator

It just so happens the important service is my splunk indexing, and I'm tracking when the queues become so full the network ports are shut off.

I found a way to do this, but it requires a set time range, because you have to hack _time a few times in the search. The idea is to turn the single "on" events into four events - the beginning and end times of the "on" status - and the "off" status just before and after the "on". Then use "connect points" to create the shape!

The idea is this:

  1. Create the transactions, or pull out some sort of events that you can create a starttime and endtime from.
  2. In this case, I used periods when the indexer was unable to receive data because its network port was closed due to over 5 minutes of blocked queues. Transaction gives me _time and _time + duration to use.
  3. Create starttime and endtime and snap them down to the 5 minute boundary.
  4. Create an earlier time by subtracting one 5-minute time period from starttime and a later time by adding one 5-minute period to endtime
  5. Create one big field that you will split into four "events" for graphing - earlier and later with 0s, and starttime and endtime with 1s.
  6. Split out the field with makemv, and explode into different events using mvexpand
  7. re-create the _time, split-by field and value of 0 or 1 for each "exploded" event
  8. Graph them, making sure to select "connect points" as the option for null values!

search:

index=_internal sourcetype=splunkd listening queues *blocked
| transaction host startswith="stopping" endswith="started"
| where duration>=300
| eval starttime=_time 
| bucket starttime span=5m
| eval earlier=starttime-300 
| eval endtime=_time+duration 
| bucket endtime span=5m
| eval later=endtime+300
| eval values=earlier + "," + host + ",0|" + starttime  +  ","  + host  + ",1|" + endtime + "," + host + ",1|" + later + "," + host + ",0"
| table values
| makemv delim="|" values
| mvexpand values
| table values
| rex field=values "(?<_time>[^,]+),(?<host>[^,]+),(?<value>[01])"
| table _time host value
| timechart span=5m max(value) by host

Click image for full size view:

Up Down Status Graphs

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...