centralized syslog server with universal forwarder

[Wahahahahaha]

We currently have a centralized syslog server in place and are looking to test Splunk in our environment.

At first, we just forwarded all our syslog messages to the Splunk server, but found that all the messages appear as if they were all coming from the same host (makes sense). So we installed a splunk universal forwarder on the syslog server in an attempt to forward the messages and keep the host information.

This worked great for Splunk, but since the forwarder instance is now listening for the syslog messages on UDP 514, my rsyslog instance no longer receives any messages for our archival logs.

What is the best way to keep our centralized syslog server in place, collecting and logging messages, while at the same time, forwarding those same messages to the Splunk server preserving the individual hosts.

Thanks in advance

[GreenKey]

Add this to your /etc/rsyslog.conf:

# must create this directory first!
$WorkDirectory /var/spool/rsyslog
$ActionQueueType LinkedList
$ActionQueueFileName splunk
$ActionQueueMaxDiskSpace 100m
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
*.* @@splunk-servername:10514

Then configure your splunk server to listen on TCP port 10514. And voila!

This uses the advanced rsyslog features of a disk-backed in-memory queue to forward all syslog message to splunk--even holding them whenever the splunk server is down!

[lguinn2]

Nice! This appears to solve the problem neatly for the test. Just make sure that when you define the TCP input in Splunk, that you specify the sourcetype=syslog (You can select it from the list.) This will ensure that Splunk extracts the host name from the syslog events, rather than using the host name of the rsyslog server.

[jbsplunk]

I would configure rsyslog write to a file, which I am presuming this is already how your instance is configured, and have Splunk monitor that file. Splunk doesn't need to listen on a port to get this data. This also give you the benefit of being able to pick up from the same place in the file as you left off if something happens to the Forwarder and it is unable to send data for some period of time. You won't be able to get splunk and rsyslog to both listen on the same port at the same time.

[lguinn2]

Splunk can monitor hundreds of files, and it's easy to set up if they are in the same directory tree. You might need to do some tuning after about 500 files - at that point, the default settings may not index the events as quickly as you'd like in Splunk. This is my preferred solution.

Are the syslog messages defined as sourcetype=syslog in Splunk? Splunk's default processing for syslog messages is to extract the host name from the syslog event - this might fix the problem in the first paragraph. You can set the sourcetype as "from list" when you set up the input.

[Wahahahahaha]

I thought about doing that, but since this is a test I was not looking to make changes to our rsyslog instance. We have a lot of filters in our rsyslog.conf file that redirect incoming messages to various files.

Once a filter is executed, it stops processing that entry. We would have to add an entry for every single filter to also go to another file that can be watched by the forwarder before we stop processing. Or we would have to have to have splunk monitor a hundreds of files (which may have to be the way we need to go)

Looking to see if there is a different option available to me.
Thanks