Getting Data In

Unable to forward syslog to third-party syslog server

michaeltay
Path Finder

I have two heavy forwarders that are responsible for sending syslog events via TCP to a third-party syslog server.

props.conf

[host::<fqdn>]
TRANSFORMS-routing = send_to_syslog

transforms.conf

[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_siem

outputs.conf

[syslog:syslog_siem]
type = tcp
server = <IP>:<port>

I am currently facing this error:

03-05-2017 00:41:43.058 +0800 ERROR DistributedClient -  Write error The operation completed successfully.
03-05-2017 00:41:43.058 +0800 ERROR OutputProc - Failed to send data to <IP>:<port>. Failed to send data with TCPClient::send. err=-3

I am 100% sure it is not a network issue. The : is actually a load balancer IP address for the syslog server.

I have tried to use the same configuration to forward to a Splunk instance, and it works beautifully.

May I know what is wrong?

0 Karma

sduff_splunk
Splunk Employee
Splunk Employee

I would check firewalls are not blocking the traffic between Splunk and your syslog server. Can you also try using udp instead of tcp as a test?

Also run

 splunk btool outputs list --debug

and confirm the outputs are being parsed correctly and there are no other config items overwritting your settings.

0 Karma

michaeltay
Path Finder

Besides forwarding the syslog events to a third-party syslog server, I am also forwarding certain syslog events to the indexer. I'm also using tcpout to send Windows event logs to port 9997 of the indexer.

transforms.conf

[send_to_both]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_siem, syslog_indexer

[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_siem

outputs.conf

[syslog]
defaultGroup = syslog_everything

[syslog:syslog_siem]
type = tcp
server = <IP>:<port>

[syslog:syslog_indexer]
type = udp
server = <IP>:<port>

[syslog:syslog_everything]

[tcpout]
defaultGroup = send_to_indexer

[tcpout:send_to_indexer]
server = <IP>:9997

[tcpout-server://<IP>:9997]
0 Karma

michaeltay
Path Finder

Hi sduff,

It's not a firewall issue, as I am able to establish connection to the load balancer via the specified TCP port.

Unfortunately, it is impossible to change the connection to UDP.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...