Splunk, VMWare, Syslog and non-default port

[mfrost8]

We're trying to setup some test monitoring of a VMWare ESX host (not ESXi). Because our Splunk instance does not run as root, I setup the UDP listener port to be something above 1024.

However, I'm not able to find anything about syslog configuration on an ESX server that shows how one might configure it to send to a remote syslog host on any port other than 514. So I don't know if it just won't work or not (the VMWare is going to try something I suggested in that regard tomorrow), but I'm not all that hopeful.

So if I have to enable a UDP listener in Splunk on port 514, I assume that means I would now have to find a way to run Splunk as root rather than the non-privileged user I'm doing this as now?

This becomes an issue because my team, who administers Splunk, are not the corporate sysadmins and as such are not given root privileges.

[Lowell · Accepted solution]

Would running a service such as syslog-ng be more acceptable to your admins? There's no reason why you can't have a standard syslog daemon listen for inbound syslog events and store them in files which splunk is configured to monitor. This is in fact a configuration that I've seen recommended by splunk a number of places.

I suspect there could be a way to do this with some kind of local firewall trick. You may be able to setup iptables (or whatever is appropriate for your system), to take traffic destined to UDP port 514 and redirect that to a local UDP port above 1024. I'm not real fluent with that kind of thing, but it seems like something like this should be possible.

[mfrost8]

The servers in question (Linux) all have syslog-ng. I like both approaches, but I suspect that asking the syadmin team to custom-configure syslog to send to us might meet with some resistance despite the fact that they don't really look at it much. Also, they have a tendency to overwrite our local configs without noticing. (fschange monitor!) I do like the sound of the firewall change though and will check that out.

Thanks Lowell and dwaddle!

[dwaddle]

The firewall trick Lowell mentions is documented in another answer here: http://answers.splunk.com/questions/6118/how-can-i-receive-syslog-udp-514-events-with-a-non-root-spl....