We have just discovered that we have lost a large amount of data. Does Splunk log when it deletes buckets? I found this question that references SPLUNK_HOME/var/log/splunk/splunkd_stdout.log, but I do not see that file on v7.0.2.
Is this something I have to turn on? Was it moved? Is there a better way?
In Alerts for Splunk Admins (splunkbase) or github version, I have alerts such as "IndexerLevel - Buckets are been frozen due to index sizing"
index=_internal sourcetype=splunkd source=*splunkd.log "BucketMover - will attempt to freeze" NOT "because frozenTimePeriodInSecs="
| rex field=bkt "(rb_|db_)(?P<newestDataInBucket>\d+)_(?P<oldestDataInBucket>\d+)"
| eval newestDataInBucket=strftime(newestDataInBucket, "%+"), oldestDataInBucket = strftime(oldestDataInBucket, "%+")
| table message, oldestDataInBucket, newestDataInBucket
Will ignore those that were frozen due to timestamp, or you could tweak that further to include those as well