I have been using syslog-ng as the log collector and Splunk as the reporter for many years and it has worked well. Many years ago, I ran into a problem with too many open file descriptors from the 32 bit syslog-ng (64 bit would have more). I had to add this to the start up script for the syslog-ng service to get past a 256 bit FD limitation:
LDPRELOAD32=/usr/lib/extendedFILE.so.1
export LDPRELOAD32
Default file descriptors increased for syslog-ng
plimit -n 65535 $$
plimit -s unlimited $$
Linux would have a different solution & 64 bit would be a better way to go.
For the host extract, I just add this to the inputs.conf file:
host_regex = ([^/]+).log$
... View more