Why is Splunk IT Service Intelligence (ITSI) not showing all entities in Service Detail View?

[paulstout]

We have 104 entities configured under a specific service in Splunk IT Service Intelligence (ITSI). This service also has 21 KPIs assigned (including Service Health Score). When in the Service Detail view, only 46 entities show when selecting the individual KPIs -- for all 21 KPIs. All 104 entities show on the Service Editor view.

I investigated the generated search for one of the KPIs and found that all 104 hosts were represented in the generated search. This is the same for KPIs generated from a base search or ad-hoc search. We do not use data models for this service.

Why could this happen? My concern is that the aggregate KPIs are not accounting for all 104 entities and our service visibility may be hindered. Any help would be greatly appreciated.

[tfletcher_splun · Accepted solution]

It means the search for the data did not return the results. This can happen if in a particular run of the KPI there just wasn't data for some of those entities.

To debug you should check for a prior run of that KPI/run the generated search from the configuration page's KPI editing modal. You want to see if the entities are represented in those search results. You want to ensure that the data is present that would be mapped to those KPIs and that it arrives in time.

One common root cause is late arriving data, in the final step of the KPI there is a section for the monitoring lag and it has a check recommended lag link. Click that link to check for late arriving data.

[paulstout]

Yikes, thanks for the answer! I actually found what was going on in our environment -- someone or something (could have been a CSV import) had configured entities in our ITSI environment that had different names, but host=xxx for one of the alias fields overlapped with the entities I'd configured. Once I removed the duplicate entities, data began reporting against all 104 hosts.

Kinda stupid of me not to check, but I just didn't think that would happen and our entities were 15,000 strong at that point.

Lesson learned and great tips for investigating future issues, thank you!