All Apps and Add-ons

Splunk Performance Graphs Not showing up.

jpewthers
Explorer

I've read the other posts on here, but they are still not helping me. None of my performance charts are showing up.

Let's start with CPU Load. Here is the service-perfdata raw line:

[SERVICEPERFDATA] 1350596518 captain-host CPU Load 0.408 0.438 OK CPU Load ok. 5s=24%;80;90;

I'm supposed to change 'avg(CpuLoad)' to something... but what? avg(CPU Load) does nothing. I've tried everything I can think of but no dice. What am I missing?

0 Karma
1 Solution

lukeh
Contributor

The first log entry in your example now has the correct format, whereas the second log entry is incorrect. If you were just providing the second log entry as a comparison then continue on, otherwise you have to delete the old template in your nagios configuration.

The nagios performance data is listed in the 'result' key/value pair, but as you are not using the CPU Nagios Plugin supported by Splunk for Nagios (ie. Check CPU Performance: http://exchange.nagios.org/directory/Plugins/System-Metrics/CPU-Usage-and-Load/Check-CPU-Performance... ) the graph is not displaying.

You can change the relevant search in the dashboard (eg. NagiosLinuxPerformanceGraphs.xml) to make the graph display correctly:

From:


 <param name="search">timechart span=5m avg(CpuSystem) as System,avg(CpuUser) as User,avg(CpuNice) as Nice,avg(CpuIowait) as Wait</param>

To:


 <param name="search">search name="CPU Load" | rex field=_raw "5s=(?.*)%" | eval total=100 | timechart span=5m avg(CpuLoad) as CpuLoad, max(total) as Total</param>

Then, remove the following lines from the same "CPU Usage" group:


<param name="charting.chart.stackMode">stacked</param>
<param name="charting.seriesColors">[0xFF0000,0x0000FF,0x00FF00,0xFF00FF]</param>
<param name="charting.legend.masterLegend"></param>

Essentially, the new search extracts a new field called 'CpuLoad' and adds a new field called 'total' which is always 100. ie.

CpuLoad=43 -> created by the rex command which matches 43 from 5s=43%

After you update the xml in the dashboard, click on the Splunk logo at the top left in Splunk for Nagios to reload the dashboards, then click on the relevant dashboard and the graph should display correctly 🙂

All the best,

Luke 🙂

View solution in original post

lukeh
Contributor

The first log entry in your example now has the correct format, whereas the second log entry is incorrect. If you were just providing the second log entry as a comparison then continue on, otherwise you have to delete the old template in your nagios configuration.

The nagios performance data is listed in the 'result' key/value pair, but as you are not using the CPU Nagios Plugin supported by Splunk for Nagios (ie. Check CPU Performance: http://exchange.nagios.org/directory/Plugins/System-Metrics/CPU-Usage-and-Load/Check-CPU-Performance... ) the graph is not displaying.

You can change the relevant search in the dashboard (eg. NagiosLinuxPerformanceGraphs.xml) to make the graph display correctly:

From:


 <param name="search">timechart span=5m avg(CpuSystem) as System,avg(CpuUser) as User,avg(CpuNice) as Nice,avg(CpuIowait) as Wait</param>

To:


 <param name="search">search name="CPU Load" | rex field=_raw "5s=(?.*)%" | eval total=100 | timechart span=5m avg(CpuLoad) as CpuLoad, max(total) as Total</param>

Then, remove the following lines from the same "CPU Usage" group:


<param name="charting.chart.stackMode">stacked</param>
<param name="charting.seriesColors">[0xFF0000,0x0000FF,0x00FF00,0xFF00FF]</param>
<param name="charting.legend.masterLegend"></param>

Essentially, the new search extracts a new field called 'CpuLoad' and adds a new field called 'total' which is always 100. ie.

CpuLoad=43 -> created by the rex command which matches 43 from 5s=43%

After you update the xml in the dashboard, click on the Splunk logo at the top left in Splunk for Nagios to reload the dashboards, then click on the relevant dashboard and the graph should display correctly 🙂

All the best,

Luke 🙂

lukeh
Contributor

You're welcome, please feel free to accept my answer 🙂

0 Karma

jpewthers
Explorer

You are a gentleman and a scholar. I think I have a better understanding of not only this app but Splunk as a whole.

jpewthers
Explorer

Many thanks for your response! It appears I should have put the line below. Here is all the data in the service-perfdata file.


1350598018 src_host="captain-host" perfdata="SERVICEPERFDATA" name="CPU Load" severity="OK" attempt="1" statetype="HARD" executiontime="0.309" latency="0.360" reason="OK CPU Load ok." result="5s=43%;80;90"

[SERVICEPERFDATA] 1350598018 captain-host CPU Load 0.309 0.360 OK CPU Load ok. 5s=43%;80;90


I tried putting 'avg(CPU Load)' in the view, since that's what I assumed the 'name' variable was, but still no dice. Any suggestions?

0 Karma

lukeh
Contributor

Hi,

the reason that the performance graphs are not showing up is because you are not using the correct template for your nagios performance data.

Please read the section entitled "Nagios Configuration (REQUIRED)" at the Splunk for Nagios page:

http://splunk-base.splunk.com/apps/22374/splunk-for-nagios

Implement steps 1 through to 5 and the graphs will display 🙂

All the best,

Luke 🙂

0 Karma

jpewthers
Explorer

I appreciate your response. I missed the most important line in that service-perfdata. I posted it below. ANy thoughts?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...