We have a deployment with approximately 500 linux systems that are sending logs via syslog on a single indexer. In some cases we notice that some logs are being lost before they arrive.
Is it possible that the indexer have a limit while processing the log flow? . Is there any best practices in this situation?
Thanks in advance.
Use a dedicated Syslog server (or servers) to centralize all your syslog traffic, write to text files and then install a universal forwarder locally to read those files.
Some advantages:
Couple of links you might find useful:
Hope that helps.
Thanks,
J
@javiergn is spot on. Here's another way of looking at the problem:
You should consider if those linux systems really need to even sent by syslog. A stronger solution would be to have a Universal Forwarder installed directly on those endpoints, directly collecting the data, also able to collect non-syslog data (like OS metrics), send the data over SSL (secure and not lossy like UDP), buffer the data when indexer not around, and scale with an increase in indexers.
"notice that some logs are being lost before they arrive" - remember that syslog is likely being sent by UDP which is lossy by nature (unlike TCP). So there's a stronger chance that the data is never arriving at your indexer. Also, if sending over UDP and your indexer is restarting, I'm pretty sure all that data is just lost.
Use a dedicated Syslog server (or servers) to centralize all your syslog traffic, write to text files and then install a universal forwarder locally to read those files.
Some advantages:
Couple of links you might find useful:
Hope that helps.
Thanks,
J
Thanks you all very much!
thanks javier for your response. Its help me a lot.
Now, I am thinking to install a heavy forwarder that acts as a syslog server, so I will send all the syslog data to it. Do you know if the heavy forwarder allows load balancing? or if it manages the buffer of events sent to the indexer.
I wouldn't install a Heavy Forwarder. See this:
https://www.splunk.com/blog/2016/12/12/universal-or-heavy-that-is-the-question.html
Either install a universal forwarder that listens to whichever Syslog ports you are using or install a dedicated Syslog server (Syslog-NG, rsyslog) that receives your traffic and writes to text files and then a UF in the same server to read those files.
Installing an HF to do this job would be like trying to kill a fly with a bazooka.