Splunk Enterprise Security

How to manage reports and alerts for 150+ indexes?

sectrainingjk
Explorer

We have a ton of indexes and need to better understand which ones have stopped receiving events so that we can report and alert on them.

We have a Splunk Enterprise v7.3.3 distributed environment with multiple (non-clustered) indexers, and non-pooled search heads configured in standalone mode. Our DSV, SH, and ES are each individual hosts and our ES is configured as a secondary SH. We manage index changes via CLI edits of indexes.conf, a deployment app, and redeployment of server classes.

We currently use the below in a dashboard panel, which generates a list of all "0-count" indexes that haven't received events in over 24 hours, but as a static list, there's a lot of additional work to get a holistic view of what's changed and when. I'd prefer query logic over a new app, as we're already hoping to pare down some of (our own) 'bloat.'

## generates a list of all "0-count" indexes that haven't received events in over 24 hours...

|tstats count where (index=* earliest=-24h latest=now()) by index

|append [|inputlookup index_list.csv |eval count=0]

|stats max(count) as count by index

|where count=0

Thanks in advance!

0 Karma

to4kawa
Ultra Champion
 This has been solved many times including:
 Meta Woot!: https://splunkbase.splunk.com/app/2949/
 TrackMe: https://splunkbase.splunk.com/app/4621/,
 Broken Hosts App for Splunk: https://splunkbase.splunk.com/app/3247/
 Alerts for Splunk Admins ("ForwarderLevel" alerts): https://splunkbase.splunk.com/app/3796/
 Splunk Security Essentials(https://docs.splunksecurityessentials.com/features/sse_data_availability/): https://splunkbase.splunk.com/app/3435/
 Monitoring Console: https://docs.splunk.com/Documentation/Splunk/latest/DMC/Configureforwardermonitoring
 Deployment Server: https://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/Troubleshootyourdeployment#Forwarde...

From @woodcock recommend

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...