Getting Data In

What is the best way to approach multiple UDP syslog Inputs to the same receiving port?

amljohnson
Explorer

There are several posts on this already (most are quite old), but I was curious how people approach multiple UDP inputs to a single UDP:514 input? I was hopeful that the following inputs.conf would perform this for me:

[udp://1.1.1.1:514]
connection_host = dns
index = index1
sourcetype = syslog

[udp://2.2.2.2:514]
connection_host = dns
index = index2
sourcetype = syslog

[udp://3.3.3.3:514]
connection_host = dns
index = index3
sourcetype = syslog

However, the UDP messages are never indexed by Splunk, despite verifying that the packets are indeed being received by the server. ONLY the 1.1.1.1 entry (in the example above) are properly indexed.

Thoughts? I'm hoping to avoid props.conf and transforms.conf if possible. Unfortunately there are limitations in quite a bit of software that will ONLY send syslog data on 514.

kabali12345
New Member

i do got same problem

0 Karma

skoelpin
SplunkTrust
SplunkTrust

You need to create a dedicated syslog server to capture your UDP traffic and log that data. You will then use a forwarder to forward that data to Splunk

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

As mentioned, dont use Splunk's UDP, use a syslog server.

But, in regards to your question, here is what is happening.

You have 3 inputs defined on the same port. Splunk wont error out on this, it will apply the configuration. However, it only applies the first one it reads. So as you mention, only the 1.1.1.1 configuration is working.

So if you cannot use a dedicated syslog server, you would need to adjust the incoming ports and redirect your sending hosts accordingly:

 [udp://1.1.1.1:514]
 connection_host = dns
 index = index1
 sourcetype = syslog

 [udp://2.2.2.2:515]
 connection_host = dns
 index = index2
 sourcetype = syslog

 [udp://3.3.3.3:516]
 connection_host = dns
 index = index3
 sourcetype = syslog

The other option, would be 3 different HF's, on different IP's listening on the same UDP/514... But thats a headache to manage...

dikshaj
Engager
0 Karma

amljohnson
Explorer

It's very unfortunate that software as flexible as Splunk requires 3rd party software to solve a problem that (at first glance) appears to be a key reason that people buy Splunk in the first place.

Are there any recommendations for a barebones syslog server that will accomplish this in a Windows environment? I'm hoping to not need to stand up a syslog server that has analysis tools that would directly compete with Splunk!

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

This isn't a real Splunk issue. It's a HA concern and issue with UDP.

Think of it this way, you need to upgrade / patch / install and app in Splunk that requires a restart. You restart. It takes 45 seconds for Splunk to restart.

That's 45 seconds that you are missing UDP syslog messages because UDP sends without waiting for an acknowledgement.

Next scenario, need to patch your windows box. Gotta drop services, patch, reboot. There goes 3 hours that you now don't have udp syslog messages coming in because Splunk was down.

Those are two of the primary scenarios of why we recommend a separate Syslog/ UDP collection method. Companies doing more then 100gb a day typically have a nix based robust syslog collection tier in place because they can't loose those logs.

Syslog for Windows.... kiwi works for low volume. Anything large scale you won't have any luck.

0 Karma

amljohnson
Explorer

Well, I would argue that if UDP HA is the concern that Splunk is trying to address with this limitation in their software, they should:

  1. Remove any ability for syslog in the first place
  2. Bundle the necessary software themselves.

Besides, wouldn't load balanced Indexers mitigate the risks in your scenario anyway? This feels more like an ongoing oversight than a feature. If you have any whitepapers or other documentation on Splunk's official stance on this I'd love to read them.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

You should create a dedicated syslog server and send the UDP traffic to the syslog server. You should then install a Universal Forwarder on the syslog server which will send data to the appropriate index

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...