Deployment Architecture

What happens to data during a network outage?

zachvida
Path Finder

I was wondering what happens to data from scripts and logs if and when a machine is either so heavily loaded it can't get enough cycles to run the process or machine lost network connectivity.

We are HPC shop, and our machine will sometime become unresponsive when running high load jobs.

Will log data just que up somewhere and get sent to the index eventually?

Tags (2)
1 Solution

araitz
Splunk Employee
Splunk Employee

Splunk will stop seeking forward in log files it is monitoring, and once the connection is restored or resources become available it will pick back up where it left off. Data that is being dropped into spool will queue. Socket inputs will start to block and there is a small queue, but otherwise they will eventually begin to drop, so it depends on the queueing mechanisms upstream and downstream as to how they will handle congestion. If Splunk cannot forward data, scripted input data will not be captured.

View solution in original post

araitz
Splunk Employee
Splunk Employee

Splunk will stop seeking forward in log files it is monitoring, and once the connection is restored or resources become available it will pick back up where it left off. Data that is being dropped into spool will queue. Socket inputs will start to block and there is a small queue, but otherwise they will eventually begin to drop, so it depends on the queueing mechanisms upstream and downstream as to how they will handle congestion. If Splunk cannot forward data, scripted input data will not be captured.

bsanjeeva
Explorer

@araitz , My question is similar to the above but I would like to know what happens to logging in below scenarios when there is an outage. I would like to know if splunk restores logging when the systems are out of outage or does it lose the logs.

1. logs are getting forwarded from an app

2.  synced from an S3 bucket

3. pulled via API

4. data coming through heavy forwarder

Thanks in Advance

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...