This question is simply out of curiosity.
If a Splunk forwarder loses its connection with its receiver (assuming there is only one receiver/no load balancing), does it hang on to the data it's supposed to forward until the connection is re-established, or are the events generated during that time lost? This might not make much of a difference for monitored files, but what about the case where you have monitored program output (i.e. running xyz program once every 60 seconds)? If the program gets run while the connection to the receiver is broken, does the output get stored until the connection is re-established?
The docs mention the use of indexer acknowledgment, but that's all assuming that a connection is available. If I'm reading the docs correctly (and I might not be), indexer acknowledgment doesn't have an effect if there's no connection at all. Specifically, it says "Without load balancing, the forwarder has no way to continue sending data if its receiving node goes down." This seems to imply that if your connection to the receiver (or all the receivers in a cluster) is unavailable, then any events generated during that time will be lost.
Any info/clarification is much appreciated!
... View more