Splunk IT Service Intelligence

Splunk IT Service Intelligence: Why is my simple glass table and deep dive not showing results?

Whistler112
New Member

Hi all,

I just got an update from our Splunk infrastructure guy that he enabled the Splunk IT Service Intelligence and I'm trying to create a very simple glass table with 1 KPI, just to try some things. I think I created the KPI correctly, it shows results when I edit it (see image) but when I place it on a glass table it says "N/A" and when I click through to the deep dive it says "No results found" . Is there someway to debug this / find out what I'm doing wrong? I tried looking through the logs exposed by the _internal index, but those aren't helping.

With regards,

Reinier

alt text

0 Karma

Whistler
Engager

Hi all, anyone else has a suggestion of what I can try?

0 Karma

jkat54
SplunkTrust
SplunkTrust

KPIs
The number displayed in a KPI tile is the number returned from the KPI search of the data. For example, you could have a KPI called Successful Logins that is a count of logins to your website. When a KPI is created in ITSI, aggregate severity-level thresholds of Normal, Low, Medium, High, and Critical are defined. If a KPI is split by entity, entity severity-level thresholds are also defined. The color corresponding to the aggregate severity-level is displayed in the KPI tile in the Service Analyzer by default. See Step 6. Set Thresholds in the ITSI Installation and Configuration manual for information about KPI severity levels.

The name of the service that the KPI is associated with is displayed on the line beneath the name of the KPI for reference.

KPI tiles that are grey indicate one of the following conditions:
The KPI search has returned no data matching the search criteria. The sparkline is flat in this case.
The KPI is associated with a disabled service (when the Show disabled service(s) check box is checked).
The KPI is associated with a service in maintenance mode (displayed in a darker shade of grey with a maintenance icon

http://docs.splunk.com/Documentation/ITSI/2.5.2/User/ServiceAnalyzer#KPIs

0 Karma

Whistler
Engager

Thanks for your answer. When I open the service analyzer, I do see my service but no KPI's (no result found). The service itself shows a flat sparkline and is grey, but not disabled or in maintenance as far as I can see.

When I click the small cog at "Top 50 KPIs" to filter KPIs, I see my service and when I select it I do see my KPI. Both the service and the KPI have their checkbox checked. So it looks like the KPI is coupled with my service.

In your previous reply (before you edited) you asked for the settings of the KPI on my glass table, do you still need that? It has thresholds set to "On". When I click on "run search" I see something strange: only 2 results. Both have an "alert_value" of "N/A". I get the same results when I edit my KPI and run the generated query. How can that be, when the preview where I can set the thresholds clearly shows 5+ values? Even when I run the query with over "all time" I only get 2 results. Both results have a 0 for field "is_service_in_maintenance" btw, so that confirms that the service isnt in maintenance.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Your KPI is driven by summary indexing. But if your root search for the KPl is only producing 2 results over time, then that data has already been summarized.

So when you're viewing your glass table for "last 15 minutes" etc, the KPI indicator is grey.

You need to change your glass table to all time to see it, but more importantly you should have more data identified by the root search. KPIs are meant for dynamic data, not 2 events from yesteryear...

0 Karma

Whistler
Engager

My root search is an ad-hoc search of: index=application_sparks sourcetype=ontwikkel errorreport | timechart count. It results in lots (971) events over the last 24 hours. As threshold field I used "count". Havent split by entity.

Calculation options can be summarized as: "Every 5 minutes take the average of count as the service/aggregate value over the last 5 minutes." I interpret that as: of all the results of the base search, pick 5 minute windows and calculate the average.

Anyway, at the end of the dialog in which I configure the KPI on the bottom it says: generated search. When I click on that I see the following query: index=application_sparks sourcetype=ontwikkel errorreport | timechart count |aggregate_raw_into_service(avg, count)|assess_severity(1cae7a64-e862-4143-a16a-69473abed004, d71f5957875be072ffdc749b, true, true)| eval kpi="SPARKS Ontwikkel - Aantal errors", urgency="5", alert_period="5", serviceid="1cae7a64-e862-4143-a16a-69473abed004" |assess_urgency``

That query is the one resulting in 2 records.

Can it be that the results of my base search arent granular enough to calculate with 5 minute windows?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...