We have upgraded Cluster Master and Cluster Peer and Search head to splunk version 7.1 . The search head shows error like below
Indexer Clustering: Search summary - AD logins by username, workstation created by jon-scheidell on the search app was deferred to run after the searchable rolling restart or upgrade is completed. There are currently 7825 deferred searches in total.
5/9/2018, 4:01:00 PM
Indexer Clustering: Search DMC Asset - Build Standalone Asset Table created by nobody on the splunk_monitoring_console app was skipped during the searchable rolling restart or upgrade.
5/9/2018, 4:01:00 PM
Other Observation :
i)none of the Scheduled searches are running.
ii) The scheduler.log on the Search head shows messages like below
INFO SavedSplunker - Skip search "DMC Asset - Build Standalone Asset Table" during searchable rolling process with nextRunTime=1525884660
INFO SavedSplunker - Defer search "summary - AD logins by username, workstation" during searchable rolling process
INFO SavedSplunker - Defer search "Summary - firewall drops by product" during searchable rolling process
INFO SavedSplunker - Defer search "Summary - WAF blocks" during searchable rolling process
INFO SavedSplunker - Defer search "Summary - Maximo ODTicketDump Deduped" during searchable rolling process
INFO SavedSplunker - Skip search "Optiv Populate Lookups - bambenekIPs - dest_ip" during searchable rolling process with nextRunTime=1525884720
…
III) The search Head’s splunkd.log file spammed with these message
CMMessages - Master Splunk version is not supported with this version of cluster master.
CMMessages - Searchable rolling restart is not supported with this version of cluster master. For more information, see the documentation on searchable rolling restart.
iv)The Cluster Master’s splunkd.log file has errors like
WARN CMMessages - Searchable rolling restart is not supported with this version of cluster master. For more information, see the documentation on searchable rolling restart.
WARN CMMessages - Master Splunk version is not supported with this version of cluster master.
INFO ClusterMasterControlHandler - Request rejected. Searchable rolling restart cannot be guaranteed if search factor=1.
