I have configured forwarding syslog to a third party device but seems the Splunk Heavy Forwarder is not forwarding the syslog. Could someone take a look at this and point out the obvious error or to a right direction please?
I can see syslog stream is coming in on UDP 514 from IP x.x.33.115 and the syslogs are appearing on the Splunk HF metric.log
psplunkls03:~ # more /data1/splunk/var/log/splunk/metrics.log | grep x.x.32.115
07-18-2016 16:09:44.426 +1000 INFO Metrics - group=per_host_thruput, series="x.x.32.115", kbps=282.791899, eps=1685.059326, kb=14139.600586, ev=84253, avg_age=6.434192, max_age=16
07-18-2016 16:09:44.427 +1000 INFO Metrics - group=udpin_connections, x.x.32.115:514, sourcePort=514, _udp_bps=226718.54, _udp_kbps=221.40, _udp_avg_thruput=228.32, _udp_kprocessed=452005.39, _udp_eps=1552.66
07-18-2016 16:10:32.426 +1000 INFO Metrics - group=per_host_thruput, series="x.x.32.115", kbps=302.518184, eps=1797.906808, kb=14520.784180, ev=86299, avg_age=6.887612, max_age=15
07-18-2016 16:10:32.426 +1000 INFO Metrics - group=udpin_connections, x.x.32.115:514, sourcePort=514, _udp_bps=262810.17, _udp_kbps=256.65, _udp_avg_thruput=228.84, _udp_kprocessed=466749.14, _udp_eps=1795.05
07-18-2016 16:11:24.425 +1000 INFO Metrics - group=per_host_thruput, series="x.x.32.115", kbps=209.181001, eps=1246.029630, kb=10877.321289, ev=64793, avg_age=10.525211, max_age=20
Configuration on Splunk Heavy Forwarder:
psplunkls03:~ # more /data1/splunk/etc/apps/search/local/inputs.conf
[splunktcp://9997]
connection_host = ip
[udp://x.x.32.115:514]
connection_host = ip
index = index_asa
sourcetype = cisco:asa
disabled = 0
…..
psplunkls03:~ # more /data1/splunk/etc/system/local/outputs.conf
[syslog]
defaultGroup=syslogGroup
[syslog:my_syslog_group]
server = 192.168.16.194:514
psplunkls03:~ # more /data1/splunk/etc/system/local/transform.conf
[send_to_mssp]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = my_syslog_group
psplunkls03:~ # more /data1/splunk/etc/system/local/props.conf
[host::x.x.32.115]
TRANSFORMS-asa = send_to_mssp
I can see Splunk Heavy Forwarder is forwarding empty packets to the designated IP 192.168.16.194 on TCP 514 though, which itself is a bit strange as it should be UDP as well right?
psplunkls03:~ # more /data1/splunk/var/log/splunk/metrics.log | grep 192.168.16.194
07-18-2016 16:32:16.341 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:33:06.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:33:58.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:34:50.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:35:28.338 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:36:00.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
... View more