Splunk unable to fetch Windows Security eventlogs

[koshyk]

We have a Windows Universal Forwader installed as service-user (svcSplunk) with read access to ALL eventlogs. (Windows 2008R2) We are getting all eventlogs except "Security" evlogs. We are struggling to find the reason for it. diags show three errors as below

    - ERROR ExecProcessor - message from "D:\SplunkUniversalForwarder\bin\splunk-winevtlog.exe" splunk-winevtlog - WinEventLogChannel::init: Init failed, unable to subscribe to Windows Event Log channel 'security': errorCode=5
   -  ERROR ExecProcessor - Couldn't start command "D:\SplunkUniversalForwarder\bin\splunk-admon.exe": The media is write protected.
   -  ERROR ExecProcessor - message from "D:\SplunkUniversalForwarder\bin\splunk-winevtlog.exe" splunk-winevtlog - WinEventLogChannel::init: Failed to bind to DC, dc_bind_time=12106 msec    

I've tested the recommendations in below URL too, but it is NOT related to Security Softwares running:
https://answers.splunk.com/answers/248673/why-is-the-splunk-universal-forwarder-on-my-domain.html

any help would be much appreciated

============ update ============
PS: (the other options/test we tried already)

• Windows Application, system eventlogs are read and working correctly. Problem is ONLY with Wineventlog:Security
• With admin permissions everything works perfect including Security logs
• No Security softwares running
• Created an interactive "test" user with same level of permissions as svcSplunk. As "test" user, eventlogs are readable

[jonny_lyse]

We had a similar problem: Splunk running as local system could not access WinEventLog:Security (But it could access "all" the other logs)

Eventually we ran "wevtutil gl security" and realised that "Local System" did not have access.

[koshyk · Accepted solution]

Our Windows Admin found the reason
This happens when your Windows Server Systems were migrated from Windows 2003 to Windows 2008

• In Win2003, there are lot of SDDL's for custom controls and eventlog access. With advent of Win2008R2, Microsoft replaced it with the 'Event Log Readers' group and group policies expected to remove the old SDDL's. However, in Win2003 it had forced it originally it was tattooed in the registry and therefore the new 'Event Log Readers' group did not appear in that SDDL
• Splunk UF was succesfully gaining access to Application and System logs due to 'Service User' (any account that has 'logon as a service' permission) being present in SDDLs, but not present in the Security log.
• The solution was to export the old SDDLs for each log and appended the access for event log readers

[dmaislin_splunk]

What happens if you run the Forwarder with domain admin permissions just as a test?

On Windows UF you should only need to changes the splunkd service account in windows services.msc and the account should have those user rights assignments :

Full control over Splunk's installation directory
Read access to any flat files you want to index
Permission to log on as a service
Permission to log on as a batch job
Permission to replace a process-level token
Permission to act as part of the operating system
Permission to bypass traverse checking

[koshyk]

with admin permissions everything works perfect. I will update this to the main query

[dmaislin_splunk]

So just work with your various permissions until you find the right settings and you should be good.

[dmaislin_splunk]

I listed the proper settings above.

[koshyk]

thanks mate. We have requested the same to Windows Admin already. I'm not sure how to verify these permissions afterwards (full control is done & verified, read access to files verified, long as service is verified). rest of the things, I'm not sure how to verify, though they said has been done.