What is the best strategy for handling Windows Event Logs in our environment?

[mkallies]

For security and audit events, we're presently planning something like this

[Everything]  --> [F5] -> [rsyslogd]  --> [splunk]

Our F5 cluster acts to provide failover. rsyslogd handles the long-term archiving, rotation, ensuring unaltered logs. We even have some reports running against it. If Splunk goes down, e.g., for maintenance, rsyslogd caches events.

The problem is that we're adding Windows events.

Our options seem to be:

• Use a forwarding app (e.g., Snare) to forward Windows Event Log Security Events via syslog (which means installing some agent on everything... which I don't like doing)
• Windows Event Log forwarding without an agent, to a Window event log receiver which would then forward without an agent. Event Log forwarding can munge logs, require group policy configuration (it's not as easy as it should be) and I don't believe provides redundancy.
• The Splunk server can be installed on Windows and retrieve Windows logs via RMI... but that means using Windows... and this solution seems to have problems scaling, so I think it's a non-option.
• Splunk forwarders can be installed (for free?) on each Windows server and forward the logs to Splunk... but if Splunk goes down, I think it might become a single point of failure. If the forwarders cache and Splunk can forward to an archival syslog server, it may be another option. I would trust the Splunk agent more than the Snare agent.

These last two solutions would mean:

[Everything]  --> [F5] -> [rsyslogd caching]  --> [splunk] --> [rsyslog archive]

What is a common practice for this?

[jnussbaum_splun]

The Universal Forwarder is the way to go. It's free, easy to install and manage (with the Deployment Server - which is also free).

You'll not only be able to collect the Win Event Logs, but other machine data from your windows endpoints using the UF - such as installed programs, listening sockets, performance metrics, registry modifications, etc. The UF will also buffer messages if a connection upstream to the indexing tier times out.

[mkallies]

If I put Splunk in front of our rsyslog archiving server, Splunk can also be set up to buffer?

Does it make sense to use some kind of Splunk forwarder rather than the "rsyslogd caching" box to receive syslog? The advantage being that syslog messages would not be lost if any device is restarted.

[jnussbaum_splun]

Well, if you were to install a UF on your windows nodes, it would make sense for the Splunk Indexers to archive the data after the indexed data reaches a certain size or age. - http://docs.splunk.com/Documentation/Splunk/6.3.2/Indexer/Automatearchiving

[mkallies]

So archiving stores events after they're outside an active window? We're not using a clustered indexer, so if the indexer is lost, we would lose the entire range of indexed Windows data.

[mkallies]

Can a heavy-forwarder be put in front of Splunk so that it receives Windows events, splitting them to Splunk (for indexing) and Syslog (for archival and alternate processing)? "Forwarding Data to a Third Party System" seems to indicate that this is possible. I'm not sure if such forwarding would impact licensing costs though...

[jnussbaum_splun]

I'd keep your syslogs going to rsyslog, and have rsyslog write them to disk where a Universal Forwarder installed on said servers will monitor and send upstream to the indexing tier.

example of writing to disk (rsyslog v3, old, I know):

$template PerHostLogSys,"/var/log/remote/sys/%HOSTNAME%/%HOSTNAME%.%$YEAR%-%$MONTH%-%$DAY%.log"
$template RemoteHostFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogfacility-text% %syslogseverity-text% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::space-cc,drop-last-lf%\n"
& ~
:inputname, isequal, "imudp" ?PerHostLogSys;RemoteHostFileFormat

Then using log rotate:
/var/log/remote/*/*/*.log {
daily
nocreate
compress
rotate=1
maxage=1
}

Hope this helps.

[mkallies]

This makes sense for syslog, but how would that archive the Windows logs?