Under Splunk DB Connect, we have data inputs created which will periodically pull data from our SQLServer database and put it into our indexes. The SQL queries used have a rising column which acts as a checkpoint column for these inputs. Things were working fine until KVStore went down on the Splunk server. To bring it up, we followed these steps which worked. Stop the search head that has the stale KV store member. Run the command splunk clean kvstore --local. Restart the search head. Run the command splunk show kvstore-status to verify synchronization. But, after doing this, Splunk DB Connect stopped writing any new data to the indexes. The DB Connect logs are filled with error that look like this for all data inputs: loading checkpoint for input title=CDA_Eportal_JobUserMapping loading checkpoint by checkpointKey=63863acecd230000be007648 from KV Store for input title=CDA_Eportal_JobUserMapping error loading checkpoint for input title=CDA_Eportal_JobUserMapping java.lang.RuntimeException: Can't find checkpoint for DB Input CDA_Eportal_JobUserMapping I am unable to even edit the data inputs to manually add checkpoints as they fail while saving. Is there any way to fix all checkpoints or clear all of them so that data gets written to the indexes again? What should I do to fix this issue?
... View more