I have configured forwarding syslog to a third party device but seems the Splunk Heavy Forwarder is not forwarding the syslog. Could someone take a look at this and point out the obvious error or to a right direction please?
psplunkls03:~ # more /data1/splunk/var/log/splunk/metrics.log | grep x.x.32.115
07-18-2016 16:09:44.426 +1000 INFO Metrics - group=per_host_thruput, series="x.x.32.115", kbps=282.791899, eps=1685.059326, kb=14139.600586, ev=84253, avg_age=6.434192, max_age=16
07-18-2016 16:09:44.427 +1000 INFO Metrics - group=udpin_connections, x.x.32.115:514, sourcePort=514, _udp_bps=226718.54, _udp_kbps=221.40, _udp_avg_thruput=228.32, _udp_kprocessed=452005.39, _udp_eps=1552.66
07-18-2016 16:10:32.426 +1000 INFO Metrics - group=per_host_thruput, series="x.x.32.115", kbps=302.518184, eps=1797.906808, kb=14520.784180, ev=86299, avg_age=6.887612, max_age=15
07-18-2016 16:10:32.426 +1000 INFO Metrics - group=udpin_connections, x.x.32.115:514, sourcePort=514, _udp_bps=262810.17, _udp_kbps=256.65, _udp_avg_thruput=228.84, _udp_kprocessed=466749.14, _udp_eps=1795.05
07-18-2016 16:11:24.425 +1000 INFO Metrics - group=per_host_thruput, series="x.x.32.115", kbps=209.181001, eps=1246.029630, kb=10877.321289, ev=64793, avg_age=10.525211, max_age=20
psplunkls03:~ # more /data1/splunk/etc/apps/search/local/inputs.conf
[splunktcp://9997]
connection_host = ip
[udp://x.x.32.115:514]
connection_host = ip
index = index_asa
sourcetype = cisco:asa
disabled = 0
…..
psplunkls03:~ # more /data1/splunk/etc/system/local/outputs.conf
[syslog]
defaultGroup=syslogGroup
[syslog:my_syslog_group]
server = 192.168.16.194:514
psplunkls03:~ # more /data1/splunk/etc/system/local/transform.conf
[send_to_mssp]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = my_syslog_group
psplunkls03:~ # more /data1/splunk/etc/system/local/props.conf
[host::x.x.32.115]
TRANSFORMS-asa = send_to_mssp
psplunkls03:~ # more /data1/splunk/var/log/splunk/metrics.log | grep 192.168.16.194
07-18-2016 16:32:16.341 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:33:06.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:33:58.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:34:50.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:35:28.338 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:36:00.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
Just a thought, but is there any reason you're not using a syslog collector (rsyslog / syslog-ng) then having Splunk read the log files? This would be the prefered option, you could also then route the syslog stream to both the local file system and to the external third party system you're aiming for.
You could try looking through this answer which details a successful config for third party routing via Splunk Heavy Forwarder: https://answers.splunk.com/answers/65818/forward-data-to-a-third-party-system.html
You're right that the data should default to UDP and NOT TCP.
Do the quantity of 'empty' forwarded packets match with the received syslog data?
On your props.conf, is [host::x.x.32.115]
are the 'x's accurate regex? Should they be '*' with escaped '.'?
I assume you have checked your host field is an IP address?