Can you configure the Receiver stanza to have a Persistent Queue?

[kuroai]

Hi All,

sorry if this is a stupid question.

When you configure a Intermediate Heavy Forwarder(Non-Indexing) receiver.
Can you add a persistent Que within this to avoid data loss for a period the upstream indexer is unavailable?

[splunktcp://9997]
persistentQueueSize=100MB <--- ???
disabled = 0

I've seen the docs and it's mostly around UF's and types of ephemeral connections, though it doesn't stipulate wither this can be done on the receiver. Though I imagine someone will tell me there's a better way to configure this.

Thanks in advance.

[adonio]

yes sir, you are on the right track

Filename: inputs.conf
Server: Intermediate forwarder
Input Stanza:
[some_input://<port_num>]
persistentQueueSize = <integer>(KB|MB|GB|TB)
* Max size of the persistent queue file on disk

The persistent queue stores the data on the disk. By default, forwarders and indexers have an in-memory input queue of 500KB.

The steps to calculate the desired size of the persistent queue:
1. Predicted Traffic in GB /Number of servers(IF|HF) = Predicted Traffic on each server in GB
2. Predicted Traffic on each server / 24 (hours in a day) = data indexed/generated in 1 hour
3. If we consider the downtime of 1 hour, the value generated above( data indexed/generated in 1 hour ) should be the size of the persistent queue.

For example, suppose the daily data traffic is 400GB and there are 2 intermediate forwarders, each will have 200GB data.
If we want to have a persistent queue for the downtime of 1 hour, the size of the persistent queue should be 200/24 = 8.33 i.e ~8 GB.

hope it helps

[kuroai]

Thanks for the detailed answer Adonio!

I'll give it ago and see how I get on.