Why are some forwarded Windows events getting dropped and I get error "Failed to get the (record id, publisher name, level id) from event..."?

[a212830]

Hi,

We have Splunk reading forwarded Windows events, and it appears to dropping events. Looking at the logs, I see the following on a semi-regular basis:

05-24-2016 15:02:54.357 -0400 WARN  WinEventLogChannel - getEventsNew: Failed to get the record id from event, channel='ForwardedEvents', 'The operation completed successfully.'.
05-24-2016 15:02:54.357 -0400 WARN  WinEventLogChannel - getEventsNew: Failed to get the publisher name from event, channel='ForwardedEvents', 'The operation completed successfully.'.
05-24-2016 15:02:54.357 -0400 WARN  WinEventLogChannel - getEventsNew: Failed to get level id from event, channel='ForwardedEvents', 'The message id for the desired message could not be found.'.

Has anyone ever seen this behavior before?

[rdjoraev_splunk]

This error message means that Splunk was unable to decode record_id from the event (unexpected type). Therefore, the root cause of this issue is specific to the file contents.

WARN WinEventLogChannel - getEventsNew: Failed to get the record id from event, channel = '\file path\ ...\filename*.evtx' 'The operation completed successfully.'

It could have a couple of possible root causes:

1) It could be specific to a file context, that needs to be reviewed and analized.

OR

2)
Make sure that you are not monitoring evtx stored files.

Event log monitor configuration values:
Windows event log (*.evt) files are in binary format. They can't be monitored like a normal text file. The splunkd service monitors these binary files by using the appropriate APIs to read and index the data within the files.

In Splunk manual on http://docs.splunk.com/Documentation/Splunk/6.4.3/Data/MonitorWindowseventlogdata
in section,
Index exported event log (.evt or .evtx) files
it states:

To index exported Windows event log files, use the instructions for monitoring files and directories to monitor the directory that contains the exported files.
and

Caution: Do not attempt to monitor a .evt or .evtx file that is currently being written to; Windows does not allow read access to these files. Use the event log monitoring feature instead.
We support indexing of .evt and .evtx files once they are exported, but do not monitor them as normal text-based logs. This explains why the file gets indexed and is not touched until splunk gets restarted.