For security and audit events, we're presently planning something like this
[Everything] --> [F5] -> [rsyslogd] --> [splunk]
Our F5 cluster acts to provide failover. rsyslogd handles the long-term archiving, rotation, ensuring unaltered logs. We even have some reports running against it. If Splunk goes down, e.g., for maintenance, rsyslogd caches events.
The problem is that we're adding Windows events.
Our options seem to be:
These last two solutions would mean:
[Everything] --> [F5] -> [rsyslogd caching] --> [splunk] --> [rsyslog archive]
What is a common practice for this?
The Universal Forwarder is the way to go. It's free, easy to install and manage (with the Deployment Server - which is also free).
You'll not only be able to collect the Win Event Logs, but other machine data from your windows endpoints using the UF - such as installed programs, listening sockets, performance metrics, registry modifications, etc. The UF will also buffer messages if a connection upstream to the indexing tier times out.
If I put Splunk in front of our rsyslog archiving server, Splunk can also be set up to buffer?
Does it make sense to use some kind of Splunk forwarder rather than the "rsyslogd caching" box to receive syslog? The advantage being that syslog messages would not be lost if any device is restarted.
Well, if you were to install a UF on your windows nodes, it would make sense for the Splunk Indexers to archive the data after the indexed data reaches a certain size or age. - http://docs.splunk.com/Documentation/Splunk/6.3.2/Indexer/Automatearchiving
So archiving stores events after they're outside an active window? We're not using a clustered indexer, so if the indexer is lost, we would lose the entire range of indexed Windows data.
Can a heavy-forwarder be put in front of Splunk so that it receives Windows events, splitting them to Splunk (for indexing) and Syslog (for archival and alternate processing)? "Forwarding Data to a Third Party System" seems to indicate that this is possible. I'm not sure if such forwarding would impact licensing costs though...
I'd keep your syslogs going to rsyslog, and have rsyslog write them to disk where a Universal Forwarder installed on said servers will monitor and send upstream to the indexing tier.
example of writing to disk (rsyslog v3, old, I know):
$template PerHostLogSys,"/var/log/remote/sys/%HOSTNAME%/%HOSTNAME%.%$YEAR%-%$MONTH%-%$DAY%.log"
$template RemoteHostFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogfacility-text% %syslogseverity-text% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::space-cc,drop-last-lf%\n"
& ~
:inputname, isequal, "imudp" ?PerHostLogSys;RemoteHostFileFormat
Then using log rotate:
/var/log/remote/*/*/*.log {
daily
nocreate
compress
rotate=1
maxage=1
}
Hope this helps.
This makes sense for syslog, but how would that archive the Windows logs?