Using Splunk indexer as a syslog forwarder

[jojoridge]

Currently we don't have any Splunk forwarders installed in our environment. We've gotten a request from the security group to see if we can forward the Syslog messages (sourced by z/Linux servers) to an ArcSight server. We still want to index the data, but would like to forward (in raw syslog format) to ArcSight. Can this be done on the Splunk indexer?

[jojoridge]

I like the syslog-ng approach, but we don't currently have any additional servers in the path between the z/Linux servers and the Splunk indexer. The infrastructure/networking guys would like to keep it that way.

Via leads generated by these responses and additional research, we appear to have arrived at a working configuration. NOTE: we haven't moved to production or tested heavily yet, but seems OK on the surface.

The 2 main references I found most helpful were:

http://wiki.splunk.com/Community:Test:How_Splunk_behaves_when_receiving_or_forwarding_udp_data

http://docs.splunk.com/Documentation/Splunk/6.0.1/Forwarding/Forwarddatatothird-partysystemsd#Send_a...

Wanted to document the changes made to hopefully assist others.

Following changes were made to Splunk UAT to test whether we can get syslog forwarding to ArcSight working:

D:\splunk\etc\system\local\props.conf (add the following at the end)

#we have 2 separate syslog inputs we'd like to forward

[source::udp:510]
TRANSFORMS-fwd2syslogout = syslogout

[source::udp:512]
TRANSFORMS-fwd2syslogout = syslogout

D:\splunk\etc\system\local\outputs.conf (add the following at the end)

# note: use the actual arcsight collector host/port below

[syslog:udpserver]

server = ARCSIGHT_CONNECTOR_HOST:ARCSIGHT_COLLECTOR_UDP_PORT

D:\splunk\etc\system\default\transforms.conf (add the following at the end)

# forward syslogs to ArcSight

[syslogout]

REGEX = .

DEST_KEY = _SYSLOG_ROUTING

FORMAT = udpserver

With all the above in place, the syslog forwarding (along with local indexing) appears to work

[mcmaster]

You can always put syslog-ng on the indexer. Have syslog-ng listen on 514 instead of splunk, write the files to a temporary directory and have splunk read those instead of listening on 514 itself. This gives you additional resiliency, as any time you restart splunk, data sent via syslog to 514 is lost. With syslog-ng, the data is still written to the disk while Splunk is restarting, and it will pick up where it left off.

[wrangler2x]

So I take it that you want to take all syslog log entries that are being received by the system running the indexer and send it also to the ArcSight server. If that is the case, I don't know how to do it with splunk, because the way the documentation looks to me is that you can send a certain subset of the syslog data somewhere else, but it does not say anything about whether or not it also indexes the data. I'm looking here:

http://docs.splunk.com/Documentation/Splunk/6.0.1/Forwarding/Forwarddatatothird-partysystemsd#Send_a...

Unless someone answers here differently or you try it yourself I'd assume you can do one or the other.

I have a similar issue. While most of my data is sent by forwarders, some is sent via syslog. And I needed to have that data also go somewhere else. As I did not have control over the environment sending me the syslog data, I came up with my own solution which might work for you.

What I do is to take their syslog data on my system running syslog-ng. Syslog-ng then sends it to two destinations:

1. Another system that I want the syslog data on.
2. To another port on my indexer that splunk listens to

This is working well. But the easiest thing would be to have the originating systems send the syslog data to both splunk and other ArcSight system if you have control over those. If not, then what I am doing is quite doable.

[mcmaster]

Using syslog-ng is definitely the most flexible option. I agree the documentation is unclear regarding whether the data is also indexed. We have in the past implemented a custom alert script that allows Splunk to selectively forward events found by a search (realtime or scheduled) via syslog.

[somesoni2]

See this.

http://answers.splunk.com/answers/5807/output-syslog-to-external-system