Getting Data In

Too many Events generated for Windows Security EventCode 4662 causing high resource issues like CPU

sat94541
Communicator

The splunk service user seems to be reading the Active Directory's "Deleted Object Container" as frequent as 60 times per second.

Every time this container is accessed, it generates Windows Security audit logs; thus increasing the volume and frequency of Windows Security logs that the Universal Forwarder has to forward for indexing. We suspect this is the cause of the issue to spiral. Also, because of the increased log volume, the Security logs now rolls every 5 hours, instead of a few also we see high resource utaliation due to

0 Karma
1 Solution

rbal_splunk
Splunk Employee
Splunk Employee
  1. At some point, unknown root cause, the service account that the SplunkForwarder runs under began accessing the Directory Service object: CN=Deleted Objects,DC=DomainDnsZones,DC=global,DC=scj,DC=loc
    1. This object does not exist.
    2. DS Access auditing was enabled on the domain controllers.
    3. This caused the object access to record a 4662 event in the event log.
    4. The service account was reading this object dozens of times per second according to the event log.
    5. The security event log on the DC was configured to roll over at 4GB size which at its shortest only held 5.5 hours of data in the 4GB of size.
    6. Hence, the events in Splunk were no newer than 5.5 hours because the security log was filling faster than the forwarder could send and index the events.
    7. Since Splunk instance is set to dump event 4662 to a null queue so it was not indexed.
    8. DS Access object auditing was disabled and event 4662 stopped being logged.
    9. Within 12 hours the forwarder caught up to the event log.
    10. DS Access object auditing was enabled again. NO 4662 events are occurring. Somewhere along the way the forwarder decided to stop attempting to access that object.

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee
  1. At some point, unknown root cause, the service account that the SplunkForwarder runs under began accessing the Directory Service object: CN=Deleted Objects,DC=DomainDnsZones,DC=global,DC=scj,DC=loc
    1. This object does not exist.
    2. DS Access auditing was enabled on the domain controllers.
    3. This caused the object access to record a 4662 event in the event log.
    4. The service account was reading this object dozens of times per second according to the event log.
    5. The security event log on the DC was configured to roll over at 4GB size which at its shortest only held 5.5 hours of data in the 4GB of size.
    6. Hence, the events in Splunk were no newer than 5.5 hours because the security log was filling faster than the forwarder could send and index the events.
    7. Since Splunk instance is set to dump event 4662 to a null queue so it was not indexed.
    8. DS Access object auditing was disabled and event 4662 stopped being logged.
    9. Within 12 hours the forwarder caught up to the event log.
    10. DS Access object auditing was enabled again. NO 4662 events are occurring. Somewhere along the way the forwarder decided to stop attempting to access that object.
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...