Getting Data In

Windows Universal Forwarder - Missing Event Logs

BP9906
Builder

I've been running a daily saved search for EventCode=1221 (Exchange space reporting).

I noticed that one of my mail serveres sometimes goes missing, but the Event does exist in the Windows Application Logs when I go to filter by 1221. I searched Splunk forwarder logs (ie 'index=_* host="mailserver"' and found many of these errors. Splunk forwarder is 4.3.4 (latest prior to 5.0 release). It seems restarting the forwarder fixes the issue for 1 day max. Then I'm back to the same issue.

Any idea whats going on here?

ERROR WinEventLogChannel - saveCheckpointStr: Failed to rename checkpoint file 'C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\persistentstorage\WinEventLog\Security_checkpoint.tmp' -> 'C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\persistentstorage\WinEventLog\Security_checkpoint': Access is denied.

drrushi_splunk
Splunk Employee
Splunk Employee

It has been observed in other cases that a possible antivirus scan may be holding the checkpoint file at the same time that Splunk is attempting to rename it.
Please stop the antivirus and retry.

Splunk 5.0.9 and 6.0 has new improvements targeting this particular scenarios, where the rename attempt will be retried once again at a later time.

0 Karma

BP9906
Builder

I've noticed that upgrading the agents to v5.0.5 it is less likely to happen. Some of my busy exchange servers are consistently sending event logs.

0 Karma

bigtyma
Communicator

I am interested if anything further was learned about this issue? Did you find a root cause?

0 Karma

RicoSuave
Builder

ERROR WinEventLogChannel - saveCheckpointStr: Failed to rename checkpoint file 'C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\persistentstorage\WinEventLog\Security_checkpoint.tmp' -> 'C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\persistentstorage\WinEventLog\Security_checkpoint': Access is denied.

This appears to be a permissions issue.

0 Karma

BP9906
Builder

5.0.1 forwarder mentions this:
INFO WinEventLogChannel - initWinEvtApi: We must be in an XP/2k3 family OS. Switching using the old Windows Event Log api: The specified module could not be found..

0 Karma

BP9906
Builder

Upgraded to 4.3.5 forwarder and 5.0.1 forwarder same issue exists.

I think that the event log monitoring stops because we're having limited disk write performance on this server. Thoughts?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...