We have a index cluster with 10+ indexers running on Splunk version 6.6.1. Some of the indexed events suddenly went missing after a network disruption (dns outage) for few minutes. There are no error messages in splunkd.log indicating any issues, replication factor and search factor are ok and all indexers are up. Events are missing in at least 2 indexes and they are recent events. All concerned indexes have sufficient retention time and the buckets haven't moved to cold storage yet. What would be the possible reason for the issue? is there a way to recover the missing events? Appreciate any pointers. ... View more

I have a similar issue as described in another question "JSON timestamps not parsed via HTTP Event Collector". But I'm seeing the issue only in a Splunk cluster setup. (Http request sending event to forwarder, then to indexer). Single node splunk instance works OK. I've also tried to use the raw endpoint as described in the answer to the above question but still doesn't work. Event timestamp (_time) is always the current time of splunk server. Is there any way to have the splunk parse the timestamp inside the event in a forwarder setup? Source type: [x_perf] DATETIME_CONFIG = INDEXED_EXTRACTIONS = json KV_MODE = NONE MAX_TIMESTAMP_LOOKAHEAD = -1 NO_BINARY_CHECK = true TIMESTAMP_FIELDS = run_timestamp TIME_FORMAT = %s TRUNCATE = 512000 category = Custom pulldown_type = 1 curl -k -u "x:UUID" https://forwarder:8088/services/collector/raw?channel=XXXXX --data-binary @json_file.json ==json file=== [ { "event": { "key1": "value1", "key2": "value2", "run_timestamp": 1513024571 }, "index": "x-index", "sourcetype": "x_perf" } ] ... View more