How do I do highly robust Splunking?

[david_lane_oe]

Hi,

I'm (we're) new to Splunk and engaging in some proof of concept work. So bear with me if this question has some bad assumptions.

I'm working in Java and have Splunk working over log4j2. So far so good. Now the trickier part: I need a specific event stream (log stream?) to be highly robust.

I'm guessing I'm going to be writing to Splunk over a network socket (so either TCP or the HTTP mechanism) and then waiting for a confirmation that the event has been committed to and processed by at least two indexers.

Can I do this (or something equivalent)?

Thanks.

[david_lane_oe]

I want the following pattern:

1) Read event from High Availability message broker.
2) [Processing goes here]
3) Log event to Splunk HA cluster
4) Receive confirmation that event has been successfully indexed (or otherwise definitely won't be going away)
5) Consume event from HA message broker, move on to next event.

Right now in my understanding there's a hole at (4). I can send the event to a forwarder, but if someone hard-resets the forwarder before it gets into Splunk then I have no way of knowing that's happened unless I use HEC acknowledgement. Is HEC acknowledgement the only way of doing this?

[david_lane_oe]

I've been assuming that the log4j2 libraries don't count as forwarders and therefore don't implement forwarder acknowledgement, and they don't seem to be heavy enough to implement HEC acknowledgement. Maybe I'm wrong?

[woodcock]

The way to do this is with useACK as documented here:
http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Protectagainstlossofin-flightdata

This will ensure that the event gets delivered (or obviously not so) to the Indexer tier. Once there, the proper thing to to is make sure that you are using a multi-site indexer cluster:

http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Multisitearchitecture

[david_lane_oe]

Thanks for your answer, I've added a comment to the question asking for a further clarification.

[rjthibod · Accepted solution]

You can get acknowledgement if you send data over HTTP(s) or if you use a forwarder.

Older ref (pre-HTTP event collector): https://answers.splunk.com/answers/221858/how-does-indexer-acknowledgement-work-with-indexer.html

HTTP event collector: http://dev.splunk.com/view/event-collector/SP-CAAAE8X

[david_lane_oe]

I'd rather not use HTTP acknowledgement right now because I'm trying to avoid adding asynchronous components to the architecture but I don't seem to have much choice as I can't see a way to get a confirmation from a forwarder back into the sending application.

[rjthibod]

Without using HTTP, posting data to Splunk is not going to have an acknowledgement to the log event generator. Without HTTP, Splunk only provides acknowledgement between forwarders and the Indexer (per the useAck link from @woodcock). You can try to add caching at the log generation / aggregation source that talks to a forwarder, but there is no specific ACK beyond what a protocol like TCP gives you.

[david_lane_oe]

Oh, well, darn. Thanks for clearing that up for me.