I have a Splunk indexer (splunk-4.0.9-74233-linux-2.6-x86_64.rpm) sending cooked data to a Splunk forwarder (active_group = Forwarder, splunk-4.2.2-101277-linux-2.6-x86_64.rpm) and from there to a syslog server (it has to be that way, don't ask).
This works, except that by the time the data is uncooked and fed to syslog, it's missing the priority, time stamp and host. I don't care about the priority, but the time stamp and host are important...
Indexer /opt/splunk/etc/system/local/outputs.conf:
[tcpout:sfwd]
server = 192.168.1.120:9997
type = tcp
sendCookedData = true
Forwarder /opt/splunk/etc/system/local/inputs.conf:
[splunktcp://:9997]
disabled = false
_SYSLOG_ROUTING = syslog_to_loghost
Forwarder /opt/splunk/etc/system/local/outputs.conf:
[syslog]
defaultGroup = syslog_to_loghost
[syslog:syslog_to_loghost]
disabled = false
server = 192.168.1.100:514
type = udp
On the syslog loghost I get the test messages I feed to the indexer as simply:
<13>payload
Where I'm assuming the 13 is the Splunk-applied priority. Where is the time stamp and hostname?
What obvious thing am I doing wrong?
In order to receive RFC3164-compliant syslog output from Splunk, you need to make sure to adequately set the timestampformat
configuration key in the [syslog:
stanza in outputs.conf, as per outputs.conf.spec:
timestampformat = <format>
* If specified, the formatted timestamps are added to the start of events forwarded to syslog.
* As above, this logic is only applied when the data is not syslog, or the syslogSourceType.
* The format is a strftime-style timestamp formatting string. This is the same implementation used in the 'eval' search command, splunk logging, and other places in splunkd.
* For example: %b %e %H:%M:%S
* %b - Abbreviated month name (Jan, Feb, ...)
* %e - Day of month
* %H - Hour
* %M - Minute
* %s - Second
* For a more exhaustive list of the formatting specifiers, refer to the online documentation.
* Note that the string is not quoted.
* Defaults to unset, which means that no timestamp will be inserted into the front of events.
[syslog:syslog_out]
server = syslog.splunk.com:514
type = tcp
timestampformat = %b %e %H:%M:%S
[syslog_test]
TRANSFORMS-routing = syslog_routing
[syslog_routing]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_out
With this configuration, all events with sourcetype "syslog_test" will be routed to syslog host "syslog.splunk.com" on port TCP/514. The default priority code of "13" will be used, which is equivalent to "user.info", and a time stamp will be added as a prefix.
[root@beefysup01 43]# cat ../sources/test.log
This is a sample syslog event
[root@beefysup01 43]# $SPLUNK_HOME/bin/splunk add oneshot ../sources/test.log -sourcetype syslog_test
Oneshot '/home/octavio/sources/test.log' added
[root@syslog.splunk.com:/]# nc -kl 514
<13>Jan 25 19:52:07 beefysup01 This is a sample syslog event
Note: I was unable to make this configuration work in versions earlier to Splunk 4.3. There are reports that this specific feature can fail in certain scenarios in 4.2.x. If you need this feature to work, please upgrade to Splunk 4.3.
This seems awfully familiar to a bug I filed a while ago, SPL-50764: "Syslog output not RFC3164 compliant (missing headers)". There are some differences in that my issue is with that Splunk isn't adding host and time in the cases where these were not supplied in the incoming message either. Still, the same bug is likely to apply to your case as well. For reference I include my bug report below.
When forwarding events from Splunk as syslog data to a syslogd, Splunk does not add headers like it should in order to be RFC3164 compliant, which goes against what is said on http://www.splunk.com/base/Documentation/latest/Admin/Forwarddatatothird-partysystems : "The forwarder sends RFC 3164 compliant events to a TCP/UDP-based server and port, making the payload of any non-compliant data RFC 3164 compliant. "
Example setup showing the issue:
Splunk instance with a raw TCP listener and a configuration to forward events received on this input as syslog data.
inputs.conf:
[tcp://4711]
connection_host = ip
sourcetype = syslogtest
props.conf:
[syslogtest]
TRANSFORMS-sl = send_to_syslog
transforms.conf:
[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_test
outputs.conf:
[syslog:syslog_test]
server = 192.168.1.4:514
In order to be RFC3164 compliant, syslog messages must include at least priority, timestamp and hostname. However, when connecting from the host "myhost" to TCP port 4711 on a Splunk server configured as in the setup above and sending a raw string such as "my test string", Splunk sends "<13>my test string" as syslog data. It should look something like "<13>Nov 9 22:01:12 myhost my test string".
Sadly no, I haven't found a workaround. It was an issue I had in a specific scenario a limited period of time, so I haven't followed up on the bug. I haven't heard of any updates though, so my guess is it is still open and unresolved.
Just found almost the same question from 2010: http://splunk-base.splunk.com/answers/8514/forward-data-to-a-syslog-server but no answer there either.
That sure sounds like what I'm seeing. Have you found a work-around? What's the bug status?