I used ngrep to look at the raw data coming into my indexer from the Universal Forwarders running on my Windows Domain Controllers. The syntax that I initially used was similar to this:
ngrep -d <interface name> -q '_MetaData:Index.main' host <DC IP>
This did not return any results. When I made the search less specific via -q '_MetaData:', I saw some data arriving with _MetaData:Index.default (which is index=main). The events all had the path _path>C:\Program Files\SplunkUniversalForwarder\bin\splunk-admon.exe . As far as I can tell, splunk-admon.exe is part of the Windows Universal Forwarder.
Since our Domain Controllers are all running Windows 2012, I decided to update "Splunk App for Windows Infrastructure" app on my SH/Indexer from version 1.0.2 to version 1.0.4, and deployed the TA-DomainController-2012R2, Splunk_TA_Windows and Splunk Add-on for Microsoft Powershell apps to my DC's via the Deployment Server.
The problem has not occurred again since I have updated these components. The ngrep search is now consistently showing raw events with the correct index metadata, like this:
_path>C:\Program Files\SplunkUniversalForwarder\bin\splunk-admon.exe.._MetaData:Index.msad
... View more