Hello,
We are working on a dashboard panel right now that is going to show overall CPU utilization for user/system but want to overlay 2 processes that we want to monitor for utilization as well. The data is coming back fine but ideally we would like the User/System graph to be one graph ie. stacked area and the process utilization to be in the form of a line chart. Any ideas?
Here is the search we are using:
index=os source=cpu host=x38dev10 | multikv fields | timechart span=60s max(pctUser) as User, avg(pctSystem) as System by host | appendcols [search index="os" sourcetype="ps" host=x38dev10 | multikv fields pctCPU, COMMAND, USER | eval process=USER."/".COMMAND | search USER=tomcat OR redis | timechart span=60s avg(pctCPU) by process useother="f"]
Here's an example :
<?xml version="1.0" encoding="utf-8"?>
<dashboard>
<label>SuperChart</label>
<row>
<chart>
<searchName>SuperChartSearch</searchName>
<title>SuperChart/title>
<!-- set up alternative palatte for stacked chart, transparancy makes the line chart more visible -->
<option name="charting.transparentBrushPalette">solidFill</option>
<option name="charting.transparentBrushPalette.colorPalette">@colorPalette</option>
<option name="charting.transparentBrushPalette.alpha">0.7</option>
<!-- set chart to use columns 0,1,2,3 where @data is the original "cube", and 0 is the time-->
<option name="charting.data1">view</option>
<option name="charting.data1.table">@data</option>
<option name="charting.data1.columns">[0,1,2,3]</option>
<option name="charting.chart.data">@data1</option>
<!-- set chart2 to use columns 0,4,5, where 0 is time, 4 + 5 are data -->
<option name="charting.data2">view</option>
<option name="charting.data2.table">@data</option>
<option name="charting.data2.columns">[0,4,5]</option>
<option name="charting.chart2.data">@data2</option>
<!-- splunk doesnt suppory 2 Y axis, so we have to clone -->
<option name="charting.axisY2">#axisY</option> <!-- clone the Yaxis into axisY2 -->
<option name="charting.axisLabelsY2">#axisLabelsY</option> <!-- clone the Yaxis Labels -->
<option name="charting.axisLabelsY2.axis">@axisY2</option> <!-- associate the new Y2 axis-labels with the Y2 axis -->
<!--create the 1st chart -->
<option name="charting.chart">column</option>
<option name="charting.chart.columnBrushPalette">@transparentBrushPalette</option>
<option name="charting.chart.stackMode">stacked</option>
<option name="charting.chart.useAbsoluteSpacing">True</option>
<option name="charting.chart.columnSpacing">20</option>
<option name="charting.chart.columnAlignment">.5</option>
<option name="charting.axisTitleY.text">CPU</option>
<option name="charting.axisLabelsY.placement">left</option>
<!--create the 2nd chart -->
<option name="charting.chart2">line</option>
<option name="charting.chart2.nullValueMode">zero</option>
<option name="charting.chart2.axisY">@axisY2</option> <!-- the Yaxis of chart 2 is a link to charting.axisY2 -->
<option name="charting.axisTitleY2">axisTitle</option>
<option name="charting.axisTitleY2.text">Memory</option>
<option name="charting.axisLabelsY2.placement">right</option>
<!-- throw the structure to layout -->
<option name="charting.layout.charts">[@chart,@chart2]</option>
<option name="charting.layout.axisLabels">[@axisLabelsX,@axisLabelsY,@axisLabelsY2]</option>
<option name="charting.layout.axisTitles">[@axisTitleX,@axisTitleY,@axisTitleY2]</option>
</chart>
</row>
</dashboard>
This assumes saved search called "SuperChartSearch" returns data like this :
... | bin _time span=1h | stats avg(User) as User avg(Sys) as Sys avg(Wait) as Wait avg(mem_free) as mem_free avg(swap_free) as swap by _time
And will display like this :
Update
If you want a realtime chart, replace the <searchName>.....</searchName>
with (for example)
<searchString>index=_internal | bin _time span=1m | stats min(eps) avg(eps) max(eps) sum(eps) perc95(eps) by _time</searchString>
<earliestTime>rt-15m</earliestTime>
<latestTime>rt</latestTime>
This is documented as a migration issue in moving from Splunk 5 to Splunk 6. In Splunk 6, simple XML uses JavaScript, not Flash. The technique used here requires Flash.
@iguinn are you saying this won't work anymore in the latest 6.3? If so how would we replicate this in 6.3? I know this is still possible because I've seen a couple folks at the conf use a similiar chart
This works brilliantly. Only problem I have is that when I try to generate a PDF, it doesn't render correctly. Anyone else having this issue?
that is surprising : )
feel free to mark the answer as accepted if its working for you.
worked like a charm. Thanks!
try something like this - I have no idea if it works - I dont have the app for *nix :
index=os host=x38dev10 source=cpu OR sourcetype=ps
| multikv
| search USER=tomcat OR USER=redis OR source=cpu
| eval metric=(if(sourcetype=="ps",USER."/".COMMAND,"CPU"))
| timechart max(pctUser) as User, avg(pctSystem) as System avg(pctCPU) as proc by metric
Thanks for the update. For some reason, I think how I have the subsearch setup with appendcols does not like realtime searches. Do I have the search setup properly for this or is there a better way to go about pulling in both the searches?
Thanks!
updated answer
This is excellent! Thank you very much for the help!
Last question though.. How would I make this into a realtime search without having to use a saved search? How would I specify in the XML to make it a realtime search if I use this as the search?
index=os source=cpu host=x38dev10 | multikv fields | timechart span=60s max(pctUser) as User, avg(pctSystem) as System by host | appendcols [search index="os" sourcetype="ps" host=x38dev10 | multikv fields pctCPU, COMMAND, USER | eval process=USER."/".COMMAND | search USER=tomcat OR redis | timechart span=60s avg(pctCPU) by process useother="f"]