Due-diligence or setup to ensure UDP syslog messages are not lost

[emille]

Hello everyone,

It's simple enough - Switches, Routers, Servers - all sending UDP syslog messages to a single point.
While most of these devices can send to multiple locations, many due to internal limitations cannot, and or only support UDP - not TCP (read: They're cheap devices)
My goal is to ensure none of these UDP syslog messages are lost in the event of Splunk or store-and-forward syslog receivers are upgrading / patching / offline for whatever reason, or reasons.

I'm reaching out to the community for feedback on how you might be dealing with this problem. I did open a case with Splunk asking for any best-practices guides on how to give the best chance that few to no logs are lost during such events, but after a couple of weeks, was advised no such guide or recommendations exist.
I'm about 50 pages in to a few forum searches for this topic. While I've found a few interesting documents that discuss handling of UDP syslog messages, nothing that really tries to approach this problem head-on.
While my search continues, I figured I'd ask.

Note: I'm not looking for a 'how-to' here - that ship has sailed. I'm now fishing for input on how you, or your organization is handling UDP Syslog messages, and or doing any due diligence to ensure they are not lost after making it to your syslog collector or splunk environment.

Any comments or ideas graciously welcomed.

[emille]

As a follow-up for this, most, if not all of our concerns are covered very well in the slide decks;
Critical Syslog Tricks (That No One Seems to Know About) Part 1 and Part 2 at the conf link you provided.

While not necessarily a best practices, they certainly cover a lot of the problems we were concerned with, as well as some 'gotchas' that we would have had to otherwise learn the hard way.

Thanks again, @burwell !

[burwell]

Hi. There were a number of great syslog / Splunk talks at the Splunk conference, .conf, in the past few years. They are best practices in action. The authors have learned and wrote up their best practices.

https://conf.splunk.com/conf-online.html?search=syslog#/

Both the slides and recordings are there. I highly recommended reading these if you haven't already.

In addition, there are two resourceful Splunk usergroup slack channels that you can join

1. rsyslog
2. syslog-ng

If you aren't a member already, follow the instructions here: https://docs.splunk.com/Documentation/Community/1.0/community/Chat

[emille]

Thanks! The information in the splunk conference videos looks extremely valuable for this topic.
I'll divert my attention to there for now.