Splunk Search

Is it possible to create one graph that has lines for each field and a timechart average for each field as additional lines?

pcorchary
Explorer

So, I have a simple search

index="prod1" source="/opt/apps/logs/my.log"  Performance Timing foobar adapter

resulting in data like below. I already have in my app the field extraction for 'foobar_call' and 'foobar_call_duration', so I don't need separate regex in the search to extract them.

What I would like to do is have one graph with multiple lines, one for each type (getfoobarPersona, validateCredential, getfoobarIdentity, etc - there are about 20 API calls here possible), and have a timechart avg() for each one as a line on the graph. Is this possible?

[c0040731][2016-10-25 00:18:04,198][foo][DEBUG][PerformanceMonitoringServiceImpl.java:48][] ,Performance Timing,com.ep.foobar.adapter.foobarAdapter,getfoobarPersona,104    
[c0040481][2016-10-25 00:18:03,895][foo][DEBUG][PerformanceMonitoringServiceImpl.java:48][] ,Performance Timing,com.ep.foobar.adapter.foobarAdapter,validateCredential,155  
[c0040731][2016-10-25 00:18:03,758][foo][DEBUG][PerformanceMonitoringServiceImpl.java:48][] ,Performance Timing,com.ep.foobar.adapter.foobarAdapter,getfoobarPersona,81 
[c0040481][2016-10-25 00:18:03,722][foo][DEBUG][PerformanceMonitoringServiceImpl.java:48][] ,Performance Timing,com.ep.foobar.adapter.foobarAdapter,getfoobarPersona,90 
[c0040731][2016-10-25 00:18:03,653][foo][DEBUG][PerformanceMonitoringServiceImpl.java:48][] ,Performance Timing,com.ep.foobar.adapter.foobarAdapter,getfoobarIdentity,100   
[c0040481][2016-10-25 00:18:03,539][foo][DEBUG][PerformanceMonitoringServiceImpl.java:48][] ,Performance Timing,com.ep.foobar.adapter.foobarAdapter,getfoobarPersona,90 
[c0040481][2016-10-25 00:18:03,449][foo][DEBUG][PerformanceMonitoringServiceImpl.java:48][] ,Performance Timing,com.ep.foobar.adapter.foobarAdapter,getfoobarPersona,94 
[c0040481][2016-10-25 00:18:03,416][foo][DEBUG][PerformanceMonitoringServiceImpl.java:48][] ,Performance Timing,com.ep.foobar.adapter.foobarAdapter,getfoobarIdentity,194   
[c0040481][2016-10-25 00:18:03,327][foo][DEBUG][PerformanceMonitoringServiceImpl.java:48][] ,Performance Timing,com.ep.foobar.adapter.foobarAdapter,getfoobarIdentity,106   
[c0040731][2016-10-25 00:18:02,960][foo][DEBUG][PerformanceMonitoringServiceImpl.java:48][] ,Performance Timing,com.ep.foobar.adapter.foobarAdapter,getfoobarPersona,103    
[c0040450][2016-10-25 00:18:02,851][foo][DEBUG][PerformanceMonitoringServiceImpl.java:48][] ,Performance Timing,com.ep.foobar.adapter.foobarAdapter,getfoobarPersona,92 
[c0040450][2016-10-25 00:18:02,720][foo][DEBUG][PerformanceMonitoringServiceImpl.java:48][] ,Performance Timing,com.ep.foobar.adapter.foobarAdapter,getfoobarIdentity,90    
[c0040763][2016-10-25 00:18:02,584][foo][DEBUG][PerformanceMonitoringServiceImpl.java:48][] ,Performance Timing,com.ep.foobar.adapter.foobarAdapter,getfoobarPersona,98
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Try something like this (you would need to fix the span of the timechart, I';m assuming the span as 1h here)

index="prod1" source="/opt/apps/logs/my.log"  Performance Timing foobar adapter | bucket span=1h _time | stats avg(foobar_call_duration) as avgduration by _time foobar_call | appendpipe [| stats avg(avgduration) as avgduration by _time | eval foobar_call="All_Svc_Average" ] | timechart span=1h avg(avgduration) by foobar_call limit=0

You'll get one line for avg foobar_call_duration for each Service (each value of field foobar_call) and one additional line for avg of all Service for that time bucket.

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try something like this (you would need to fix the span of the timechart, I';m assuming the span as 1h here)

index="prod1" source="/opt/apps/logs/my.log"  Performance Timing foobar adapter | bucket span=1h _time | stats avg(foobar_call_duration) as avgduration by _time foobar_call | appendpipe [| stats avg(avgduration) as avgduration by _time | eval foobar_call="All_Svc_Average" ] | timechart span=1h avg(avgduration) by foobar_call limit=0

You'll get one line for avg foobar_call_duration for each Service (each value of field foobar_call) and one additional line for avg of all Service for that time bucket.

0 Karma

pcorchary
Explorer

Wow! That's bloody awesome! Thank you so much!

Side question ... is there a way in the search terms to tell the graph to be line type instead of the default?

0 Karma

gokadroid
Motivator

Depending on, when you are charting the values which one out of max or min you want to take as the response time values (given timechart will automatically aggregate results) you can try this:

| timechart max(foobar_call_duration) as callRespTime, avg(foobar_call_duration) as callAvgResp by foobar_call

Or

| timechart min(foobar_call_duration) as callRespTime, avg(foobar_call_duration) as callAvgResp by foobar_call

Or

| timechart values(foobar_call_duration) as callRespTime, avg(foobar_call_duration) as callAvgResp by foobar_call

Whichever suits your need

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...