Splunk Search

Sending Syslog to a Third Party Is Not Working

phil_wang
New Member

I have configured forwarding syslog to a third party device but seems the Splunk Heavy Forwarder is not forwarding the syslog. Could someone take a look at this and point out the obvious error or to a right direction please?

  1. I can see syslog stream is coming in on UDP 514 from IP x.x.33.115 and the syslogs are appearing on the Splunk HF metric.log

psplunkls03:~ # more /data1/splunk/var/log/splunk/metrics.log | grep x.x.32.115
07-18-2016 16:09:44.426 +1000 INFO Metrics - group=per_host_thruput, series="x.x.32.115", kbps=282.791899, eps=1685.059326, kb=14139.600586, ev=84253, avg_age=6.434192, max_age=16
07-18-2016 16:09:44.427 +1000 INFO Metrics - group=udpin_connections, x.x.32.115:514, sourcePort=514, _udp_bps=226718.54, _udp_kbps=221.40, _udp_avg_thruput=228.32, _udp_kprocessed=452005.39, _udp_eps=1552.66
07-18-2016 16:10:32.426 +1000 INFO Metrics - group=per_host_thruput, series="x.x.32.115", kbps=302.518184, eps=1797.906808, kb=14520.784180, ev=86299, avg_age=6.887612, max_age=15
07-18-2016 16:10:32.426 +1000 INFO Metrics - group=udpin_connections, x.x.32.115:514, sourcePort=514, _udp_bps=262810.17, _udp_kbps=256.65, _udp_avg_thruput=228.84, _udp_kprocessed=466749.14, _udp_eps=1795.05
07-18-2016 16:11:24.425 +1000 INFO Metrics - group=per_host_thruput, series="x.x.32.115", kbps=209.181001, eps=1246.029630, kb=10877.321289, ev=64793, avg_age=10.525211, max_age=20

  1. Configuration on Splunk Heavy Forwarder:

psplunkls03:~ # more /data1/splunk/etc/apps/search/local/inputs.conf
[splunktcp://9997]
connection_host = ip

[udp://x.x.32.115:514]
connection_host = ip
index = index_asa
sourcetype = cisco:asa
disabled = 0
…..

psplunkls03:~ # more /data1/splunk/etc/system/local/outputs.conf
[syslog]
defaultGroup=syslogGroup

[syslog:my_syslog_group]
server = 192.168.16.194:514

psplunkls03:~ # more /data1/splunk/etc/system/local/transform.conf
[send_to_mssp]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = my_syslog_group

psplunkls03:~ # more /data1/splunk/etc/system/local/props.conf
[host::x.x.32.115]
TRANSFORMS-asa = send_to_mssp

  1. I can see Splunk Heavy Forwarder is forwarding empty packets to the designated IP 192.168.16.194 on TCP 514 though, which itself is a bit strange as it should be UDP as well right?

psplunkls03:~ # more /data1/splunk/var/log/splunk/metrics.log | grep 192.168.16.194
07-18-2016 16:32:16.341 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:33:06.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:33:58.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:34:50.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:35:28.338 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:36:00.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00

0 Karma

hmclaren_splunk
Splunk Employee
Splunk Employee

Just a thought, but is there any reason you're not using a syslog collector (rsyslog / syslog-ng) then having Splunk read the log files? This would be the prefered option, you could also then route the syslog stream to both the local file system and to the external third party system you're aiming for.

You could try looking through this answer which details a successful config for third party routing via Splunk Heavy Forwarder: https://answers.splunk.com/answers/65818/forward-data-to-a-third-party-system.html

You're right that the data should default to UDP and NOT TCP.

Do the quantity of 'empty' forwarded packets match with the received syslog data?

On your props.conf, is [host::x.x.32.115]are the 'x's accurate regex? Should they be '*' with escaped '.'?
I assume you have checked your host field is an IP address?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...