Getting Data In

How to avoid / delete duplicate events using routers logging to central syslog

sonicZ
Contributor

Currently we are logging all our network device data from our routers to a single syslog host.
This syslog host forward to a central syslog logger which our splunk indexer monitors directly.

However we would like to log to multiple syslog hosts from the routers instead of just one but this would cause a lot of duplicate entries in our central syslogger. Anyone have a good approach to handle routers logging to multiple syslog hosts(for redundancy) but filtering duplicates before they index into the Splunk indexers?

Would rather not just pipe to dedup

results | dedup 

hopefully there is a solution to throw away dupes or an entirely new approach.

1 Solution

dwaddle
SplunkTrust
SplunkTrust

Well, Splunk itself isn't going to be able to know those events coming from different syslog servers are actually duplicates. So, there's no real way (within Splunk) to avoid the duplication.

One viable alternative is to cluster your syslog servers - use a floating IP address between the two (Red Hat's piranha / pulse comes to mind) and send all of your log data to the floating IP. Then you keep your high availability, but with only one copy of each event.

View solution in original post

dwaddle
SplunkTrust
SplunkTrust

Well, Splunk itself isn't going to be able to know those events coming from different syslog servers are actually duplicates. So, there's no real way (within Splunk) to avoid the duplication.

One viable alternative is to cluster your syslog servers - use a floating IP address between the two (Red Hat's piranha / pulse comes to mind) and send all of your log data to the floating IP. Then you keep your high availability, but with only one copy of each event.

dwaddle
SplunkTrust
SplunkTrust

I've never used haproxy/keepalived but I think that for practical matters here they'd function similarly.

0 Karma

sonicZ
Contributor

Thanks for the info Dwaddle, We were thinking of using haproxy and keep aliveD on two different syslog servers basically doing a software VIP load balanced. I'll check into piranha / pulse too.

0 Karma

jgedeon120
Contributor

You can collect your logs to as many syslog servers and have those send to a central syslog server then have the central syslog server send to Splunk. Syslog-ng is very configurable.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...