Deployment Architecture

How to debug why a change in the Search Head Cluster Captain causes summary indexing jobs to look for data in the past?

nwales
Path Finder

We have an every minute summary indexing job which runs happily looking at data from a few minutes ago.

If we push a config out causing a captain change, we have seen twice in a row now that the new captain is looking at data 4 days ago. If we restart that instance and the original instance takes over, then it goes back to looking at the present time.

Last time this happened, we had to cycle through all three instances before we got back to the original. The second instance was running about 6 minutes behind, which, while not as bad, causes us duplicate information.

SH1 = currenty
SH2 = -6m
SH2 = -4days

The summary job does the right thing and attempts to backfill the data, but clearly this is not the behavior we are looking for.

What should I be looking for to start debugging this?

0 Karma

gustavomichels
Path Finder

What version are you using? 6.2.4 included a fix for summary searches (SPL-99279 - http://docs.splunk.com/Documentation/Splunk/6.2.4/ReleaseNotes/6.2.4) which might be related.

nwales
Path Finder

That might help, we've seen that issue separately but if the scheduler has been tightened up then it might help here too.

Currently we're on 6.2.3

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...