Hello Malmoore and thank you for the response.
The DC's are a mix of 2008R2 and 2012R2
The ASA (Cisco Adaptive Security Appliance [Firewall]) is only able to send syslog messages to an IP on the inside interface. Therefore in 4 of the 5 sites the ASA directs the syslog messages to the local DC running Splunk Forwarder. Packet captures show the logs arrive and Netstat confirms Splunk.exe (forwarder) is listening on the port. This configuration is working for all servers running 2008R2. The one server running 2012R2 does not appear to forward the syslog events from the ASA (nor is it any longer sending Windows logs), however I suspect the problem lies with Splunk Light not logging them in the DB (ignoring them). The 2012R2 server was sending Windows logs at one time but these are no longer coming through.
We are only interested in the security logs on the DC's. Specifically we are interested to know when a user logs onto their local machine. Only the local DC logs this event so we need the events from all DC's.
DC's not sending Windows logs:
2 server running 2008R2 (both were sending logs at one time but stopped 1 week ago - no changes were implemented at that time - we are reinstalling forwarder again to try to bring it back online.)
1 server running 2012R2 (was sending logs at originally but stopped after a reinstall of the forwarder while troubleshooting ASA logs not coming through)
DC's not forwarding ASA logs
Same 2012R2 server no longer sending security logs. Splunk light has never registered the IP of the ASA as a source. The IP has appeared in the Hosts list automatically for the other firewalls. Packet capture shows the same syslog entries leaving the forwarder destined for the Splunk server.
Searching for DC's by hostname returns entries from the security logs from that host (results returned for all hosts - not all of them have recent events).
In troubleshooting we had tried to get all AD events to see if that would work and in many cases that allowed events to start flowing. As a result some servers are sending more information than desired. We attempted to change this under Home, Add Data, forward, Existing, (select server class for windows servers [all 5 DC's are listed as forwarders]), Next, Local Event Logs, Security, Review, Submit. However the Sources and Sourcetypes lists on the home page indicates last update for WinEventLog:Security 1/25. Additionally this seems to stop all Windows logs coming in including Security logs.
I suspect this to be a key part of the problem. Splunk seems to be ignoring logs due to some setting in the Add Data area or elsewhere. We have tried so many different combinations and are at a loss as to what may produce the expected result. We have run packet captures on one forwarder not showing Windows logs in Splunk and see what look like Windows event logs being sent from the forwarder yet not appearing in the DB.
Config from: C:\Program Files\SplunkUniversalForwarder\etc\system\local
deploymentclient
[target-broker:deploymentServer]
targetUri = (IP of Splunk Server):8089
inputs
[default]
host = B-DC01
[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0
outputs
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = (Splunk IP):9997
[tcpout-server://(Splunk IP):9997]
Server
[general]
serverName = B-DC01
pass4SymmKey = (redacted)
[sslConfig]
sslKeysfilePassword = (redacted)
[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder
[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free
Thank you.
... View more