Run splunk as non root user :
http://docs.splunk.com/Documentation/Splunk/5.0/installation/RunSplunkasadifferentornon-rootuser
Solution 0 : run splunk as root
Solution 1 : redirect the port
Use iptables to redirect the port 514 to another range, like 10514.
The iptable rules are stored in /etc/sysconfig/iptables. It contains the rules which will be loaded during bootup.
The reason why the rules disappeared after reboot is that, they were not saved to this file.
You need to add the rules using the following commands:
iptables -t nat -A PREROUTING -p UDP -m udp --dport 514 -j REDIRECT --to-ports 10514
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 514 -j REDIRECT --to-ports 10514
iptables-save
and then, pipe the output from iptables-save into this file.
iptables-save > /etc/sysconfig/iptables
Alternatively, you could also copy the iptables file from/etc/sysconfig from any other working SSIM which already
has the rules for syslog redirect.
And setup splunk to listen to this new port
my favorite config for UDP is
[udp://10514]
sourcetype=syslog
connection_host=ip
# do not resolve the hostname, it is usually included in the syslog events
queueSize=1MB
# to add some buffer in case of indexer slow
persistentQueueSize = 5MB
# to add some disk buffer too
Solution 2 : use a syslog server
- setup a rsyslog/ syslog-ng server to run as system process, and listen to 514
- have the logs being written to disk, by example a folder per host, with log rotation.
- have splunk monitor those folders.
the advantage if this method is that UDP is non resilient and volatile, so the syslog server will act as a file buffer, and if Splunk is restarting it will catch up once up.
Another example of Solution 1 is doing this redirect from 514 to 5514 :
Poke hole in iptables to allow ports.
Redirect for port 515 up to 5514 which we are listening on.
iptables -I INPUT -p tcp --dport 5514 -j ACCEPT
iptables -t nat -A PREROUTING -d MY.IP -p tcp -m tcp --dport 514 -j DNAT --to-destination MY.IP:5514
iptables -t nat -A PREROUTING -d MY.IP -p udp -m udp --dport 514 -j DNAT --to-destination MY.IP:5514
Be sure to "service iptables save" after modifying iptables, or modify etc/sysconfig/iptables directly.
Run splunk as non root user :
http://docs.splunk.com/Documentation/Splunk/5.0/installation/RunSplunkasadifferentornon-rootuser
Solution 0 : run splunk as root
Solution 1 : redirect the port
Use iptables to redirect the port 514 to another range, like 10514.
The iptable rules are stored in /etc/sysconfig/iptables. It contains the rules which will be loaded during bootup.
The reason why the rules disappeared after reboot is that, they were not saved to this file.
You need to add the rules using the following commands:
iptables -t nat -A PREROUTING -p UDP -m udp --dport 514 -j REDIRECT --to-ports 10514
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 514 -j REDIRECT --to-ports 10514
iptables-save
and then, pipe the output from iptables-save into this file.
iptables-save > /etc/sysconfig/iptables
Alternatively, you could also copy the iptables file from/etc/sysconfig from any other working SSIM which already
has the rules for syslog redirect.
And setup splunk to listen to this new port
my favorite config for UDP is
[udp://10514]
sourcetype=syslog
connection_host=ip
# do not resolve the hostname, it is usually included in the syslog events
queueSize=1MB
# to add some buffer in case of indexer slow
persistentQueueSize = 5MB
# to add some disk buffer too
Solution 2 : use a syslog server
- setup a rsyslog/ syslog-ng server to run as system process, and listen to 514
- have the logs being written to disk, by example a folder per host, with log rotation.
- have splunk monitor those folders.
the advantage if this method is that UDP is non resilient and volatile, so the syslog server will act as a file buffer, and if Splunk is restarting it will catch up once up.
For ubuntu , save iptables persistently using
sudo apt-get install iptables-persistent
sudo /etc/init.d/iptables-persistent save # For Saving new rules
sudo /etc/init.d/iptables-persistent reload # For reloading
check setup is good by
sudo iptables -L -vt nat
PS : another solution is to use a proxy (like nginx) to redirect ports to splunk (with splunk using ports > 1024)
It's the same method for udp/tcp, or even for splunkweb port.
For whatever it's worth, in 6.1.1 the same limitation still exists. I was hoping that because 6.1 now starts as root and switches to a named (non-privileged) user during startup that this limitation may have been lifted, but testing shows otherwise.