How to troubleshoot why events of the same sourcetype are being indexed in two indexes?

[psharkey]

I have Splunk Universal Forwarders installed on my Windows Domain Controllers. Up until 5 weeks ago, sourcetype=ActiveDirectory events were exclusively being indexed in an index named msad.

Starting 5 weeks ago, some of the sourcetype=ActiveDirectory events have been indexed in the default index (main). The DC's that have indexed some sourcetype=ActiveDirectory events in index=main have also indexed other sourcetype=ActiveDirectory events in index=msad.

For what it is worth, there are four domain controllers, three of which are running Splunk Universal Forwarder version 6.1.3 and the other is running version 5.0.4. The DC running UF version 5.0.4 has consistently indexed sourcetype=ActiveDirectory events in index=msad if that matters.

The inputs.conf on my indexer routes these sourcetypes to index=msad, so I am curious to know why/how some of the events are winding up in main. Any help would be appreciated.

[psharkey · Accepted solution]

I used ngrep to look at the raw data coming into my indexer from the Universal Forwarders running on my Windows Domain Controllers. The syntax that I initially used was similar to this:

ngrep -d <interface name> -q '_MetaData:Index.main' host <DC IP>

This did not return any results. When I made the search less specific via -q '_MetaData:', I saw some data arriving with _MetaData:Index.default (which is index=main). The events all had the path _path>C:\Program Files\SplunkUniversalForwarder\bin\splunk-admon.exe. As far as I can tell, splunk-admon.exe is part of the Windows Universal Forwarder.

Since our Domain Controllers are all running Windows 2012, I decided to update "Splunk App for Windows Infrastructure" app on my SH/Indexer from version 1.0.2 to version 1.0.4, and deployed the TA-DomainController-2012R2, Splunk_TA_Windows and Splunk Add-on for Microsoft Powershell apps to my DC's via the Deployment Server.

The problem has not occurred again since I have updated these components. The ngrep search is now consistently showing raw events with the correct index metadata, like this:

_path>C:\Program Files\SplunkUniversalForwarder\bin\splunk-admon.exe.._MetaData:Index.msad