Deployment Architecture

Missing events after network disruption

thol
Explorer

We have a index cluster with 10+ indexers running on Splunk version 6.6.1. Some of the indexed events suddenly went missing after a network disruption (dns outage) for few minutes. There are no error messages in splunkd.log indicating any issues, replication factor and search factor are ok and all indexers are up.

Events are missing in at least 2 indexes and they are recent events. All concerned indexes have sufficient retention time and the buckets haven't moved to cold storage yet.

What would be the possible reason for the issue? is there a way to recover the missing events?
Appreciate any pointers.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi thol,
some additional information:

  • how do you receive events, by syslog or by Universal Forwarder?
  • if syslog, the network problem you said was between source and Splunk receivers?
  • if syslogs, how do you receive them, using one or more Heavy forwarders? have you a Load Balancer?
  • how is connected your storage to Indexers, NAS?

Bye.
Giuseppe

thol
Explorer

Thank you Giuseppe,

  • Events are received through Http Event Collector from a heavy forwarder. Event was already in the index and the events were already seen from the dashboard.
  • During network outage. Its possible some indexers were not able communicate with index master or peers for a few minutes.
  • all storage to indexers are local disks.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...