Splunk Search

Why is scheduler failing after upgrading a search head pool?

ekost
Splunk Employee
Splunk Employee

Our search head pool nodes were recently upgraded from 6.6.1 to 7.3.0. After the upgrade, the scheduled searches have failed, breaking the reports and alerts. The log shows lots of empty warning messages:

12-24-2019 23:59:59.227 +0100 WARN SavedSplunker -

So DEBUG logging was enabled and we see messages like:

12-24-2019 23:59:59.226 +0100 DEBUG SavedSplunker - savedSearchUpdated field changed: savedsearch_id="admin;search;InternalSplunkdLogAlert", field_name="__ss_type", old_val="scheduled", new_val=""
12-24-2019 23:59:59.225 +0100 DEBUG SavedSplunker - AlertNotifier queued notifications=0, managedSearchCount=0, managedSchedulerSearchCount=0, managedSchedulerRTSearchCount=0
12-24-2019 23:59:59.224 +0100 DEBUG SavedSplunker - lock file already exists, search head skipping execution of: savedsearch_id="nobody;splunk_monitoring_console;DMC Asset - Build Standalone Asset Table", now=1562854500, 

The Search Head Pooling feature was deprecated for some time, but there was no mention of a bug or behavior change that I could find. What happened?

ekost
Splunk Employee
Splunk Employee

This is the known issue SPL-173038 that appears in Splunk 7.1.x, 7.2.x and 7.3.x effecting the Search Head Pooling (SHP) feature and the scheduler. No scheduled searches or alerts will run if SHP nodes are upgraded to 7.1.x and later. The only remediation is to switch to search head clustering. The search head pooling feature was removed in Splunk Enterprise 8.0.0.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...