Splunk IT Service Intelligence

ITSI Thresholds vs Public Holidays

irwazr
New Member

Hi everyone,

We are using ITSI to display Glass Table dashboards for staff to view the health of critical systems. We have thresholds set for Weekdays and Weekends. Obviously the thresholds for weekends are significantly lower than weekdays.

This creates a headache when a public holiday rolls around however, as we see weekend traffic volumes on a weekday, and so the dashboards begin to light up like a Christmas Tree. Hence we're forced to change the weekday thresholds to accept service levels as low as a weekend, which is far from ideal and hinders the tool from early detection and alerting.

Has anyone found a way to manage public holiday thresholds?

Thanks.

0 Karma

irwazr
New Member

I've been trying out options as a workaround, the most suggested of which is the use of a lookup table. However the problem with the lookup is what to do when we know a search result falls on a public holiday. When a result is on a public holiday, you would assume we would exclude/ignore the event. This would produce a null result for the KPI. The biggest problem with this is that a lack of events returned is also a platform level indexing issues that are possible within Splunk... so public holidays and index performance issues would trigger the same KPI thresholds as each other (i.e. null = 0, and 0 equals bad).

Given ITSI is a tool aimed at service health monitoring, and that services are directly impacted by public holidays, it would seem this is critical functionality currently missing from the product.

0 Karma

esnyder_splunk
Splunk Employee
Splunk Employee

In a future release we hope to have a "Special Days" feature that takes care of thresholds for special days like Black Friday, Christmas, and other significant days for your organization. However, this functionality currently does not exist 😞

Keep your eye on the new features lists for each release and hopefully it'll be there soon: https://docs.splunk.com/Documentation/ITSI/latest/ReleaseNotes/Newfeatures

jaime_ramirez
Communicator

Have you tried using Maintenance Windows as a workaround ? Also you could try adjusting your thresholds using lookups for Holidays (we have solved it this way).

Cheers!!!

0 Karma

irwazr
New Member

The problem with a maintenance window is that it hides the problem. It simply suppresses the alert and leaves everyone blind to service availability on what is still an important day of operation for the organisation (i.e. volumes may be lower, but criticality of services are just as high).

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...