How to troubleshoot why I am not getting any data in the Splunk App for Active Directory?

[kpavan]

Hi All!

My issue is I am not able to get the data in Splunk App for Active Directory (Topology, controllers etc). Below are the details which I have done so for.

1. Installed Enterprise Splunk full 6.3.2 (i.e 60 days) on Redhat Linux.
2. Configured receiving port 9997
3. Installed Splunk Universal Forwarder on Windows 2008 R2 DC
4. Configured as per the on both receiving and forward side http://docs.splunk.com/Documentation/ActiveDirectory/1.2.2/DeployAD/AbouttheSplunkAppforActiveDirect...
5. Nothing changed on UF, only changed index name from default index and same as been changed in receiving end indexes.conf file as well

6. SA_ldapsearch; ldap.conf configured and tested connection and successful

   [default]
   alternatedomain = splunk.local
   basedn = dc=splunk,dc=local
   binddn = CN=Administrator,CN=Users,DC=splunk,DC=local
   port = 3268
   server = xx.xx.xx.xx
   ssl = 0

7. When I search AD data like index=myadindex | stats count by myadindex am able to see the logs which are coming from the AD

8. But when I check Splunk AD App topology view or domain stats, there were no result found in the app page.

9. I did check

   domain-list|dedup host|outputlookup DomainList.csv
   and

   domain-selector-search|outputlookup DomainSelector.csv
   but there were no results returned.

   1. FYI... Couple of things to know why on SplunkUF splunkd.log as below,

   01-14-2016 04:56:52.189 -0500 INFO TailReader - Registering metrics callback for: batchreader0
   01-14-2016 04:56:52.189 -0500 INFO TailReader - Starting batchreader0 thread
   01-14-2016 04:56:52.938 -0500 INFO TcpOutputProc - Connected to idx=192.168.18.206:9997
   01-14-2016 04:57:05.028 -0500 INFO TailReader - Could not send data to output queue (parsingQueue), retrying...
   01-14-2016 06:01:38.000 -0500 WARN ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-regmon.exe"" ConfMetrics - single_action=BASE_INITIALIZE took wallclock_ms=1327
   01-14-2016 06:02:19.652 -0500 WARN ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-regmon.exe"" ConfMetrics - single_action=BASE_INITIALIZE took wallclock_ms=1670
   01-14-2016 06:02:56.795 -0500 INFO TailReader - ...continuing.
   01-14-2016 06:03:06.826 -0500 INFO TailReader - Could not send data to output queue (parsingQueue), retrying...
   01-14-2016 06:03:16.857 -0500 INFO TailReader - ...continuing.
   01-14-2016 06:03:27.106 -0500 INFO TailReader - Could not send data to output queue (parsingQueue), retrying...
   01-14-2016 06:03:37.137 -0500 INFO TailReader - ...continuing.
   01-14-2016 06:04:01.426 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\btool.log'.
   01-14-2016 06:04:01.426 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\mongod.log'.
   01-14-2016 06:04:01.426 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\scheduler.log'.
   01-14-2016 06:06:36.491 -0500 WARN ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-regmon.exe"" ConfMetrics - single_action=BASE_INITIALIZE took wallclock_ms=1685
   01-14-2016 06:06:47.301 -0500 INFO TailReader - Could not send data to output queue (parsingQueue), retrying...
   01-14-2016 06:07:02.340 -0500 INFO TailReader - ...continuing.
   01-14-2016 06:08:38.124 -0500 WARN ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-regmon.exe"" ConfMetrics - single_action=BASE_INITIALIZE took wallclock_ms=2606
   01-14-2016 06:12:56.835 -0500 INFO TailReader - Could not send data to output queue (parsingQueue), retrying...
   01-14-2016 06:13:01.842 -0500 INFO TailReader - ...continuing.
   01-14-2016 06:13:59.017 -0500 INFO TailReader - Could not send data to output queue (parsingQueue), retrying...
   01-14-2016 06:14:39.077 -0500 INFO TailReader - ...continuing.
   Please help with fixing issue.

Thanks in advance!

[ppablo]

Hi @kpavan

I noticed you linked to this documentation for the "Splunk App for Active Directory (Legacy)", but you said you're using Splunk 6.3.2. The first section on the documentation you linked, it says:

" If you currently run the Splunk App for Active Directory on Splunk Enterprise 6.x and later, you can install the Splunk App for Windows Infrastructure onto the same Splunk instance as the existing Splunk App for Active Directory. The Splunk App for Windows Infrastructure allows you to configure it to view and display the data you have already collected with the Splunk App for Active Directory. Once you have confirmed that this app sees all your data, you can delete the older apps."

So, you shouldn't be using the Splunk App for Active Directory unless you're on Splunk 5.x or below. You should install and configure the Splunk App for Windows Infrastructure instead:
https://splunkbase.splunk.com/app/1680/

[kpavan]

Hi ppablo,

Thanks for addressing the issue!

I have installed Splunk app for windows infrastructure, but after install in configuration check data tap getting below error

Data from Splunk Add-on for Microsoft Windows

All searches have completed
No data detected: Please make sure Splunk Forwarders are properly configured and sending data
WARNING: Search "sourcetype="Perfmon*" | head 5" did not return any events in the last 24 hours
WARNING: Search "sourcetype="WinHostMon*" | head 5" did not return any events in the last 24 hours
WARNING: Search "sourcetype="WinPrintMon*" | head 5" did not return any events in the last 24 hours
WARNING: Search "sourcetype="WinRegistry*" | head 5" did not return any events in the last 24 hours
WARNING: Search "sourcetype="WMI*" | head 5" did not return any events in the last 24 hours
WARNING: Search "sourcetype="WinEventLog*" OR sourcetype="XmlWinEventLog*" | head 5" did not return any events in the last 24 hours

Data from Splunk Add-on for Microsoft Windows Active Directory
Critical data could not be found
No data detected: Please make sure Splunk Forwarders are properly configured and sending data
ERROR: Search "sourcetype="MSAD*" | head 5" did not return any events in the last 24 hours
ERROR: Search "sourcetype="ActiveDirectory*" | head 5" did not return any events in the last 24 hours
WARNING: Search "sourcetype="WinEventLog*" OR sourcetype="XmlWinEventLog*" | head 5" did not return any events in the last 24 hours
WARNING: Search "sourcetype="Perfmon*" | head 5" did not return any events in the last 24 hours

But, if I search with query index=ad-* sourcetype=* | stats count by sourcetype am getting the result with counts. One thing need to know do I need to specify configs if sourcetypes above (sourcetype="Perfmon* etc) sending data use the index as (ad-perfmon something like that)?

Below are the indexex.conf for each app on receiving side

Splunk_for_ActiveDirectory
[ad-msad]
homePath = $SPLUNK_DB/ad-msad/db
coldPath = $SPLUNK_DB/ad-msad/colddb
thawedPath = $SPLUNK_DB/ad-msad/thaweddb
maxDataSize = 10000
maxHotBuckets = 10

[ad-perfmon]
homePath = $SPLUNK_DB/ad-perfmon/db
coldPath = $SPLUNK_DB/ad-perfmon/colddb
thawedPath = $SPLUNK_DB/ad-perfmon/thaweddb
maxDataSize = 10000
maxHotBuckets = 10

[ad-winevents]
homePath = $SPLUNK_DB/ad-winevents/db
coldPath = $SPLUNK_DB/ad-winevents/colddb
thawedPath = $SPLUNK_DB/ad-winevents/thaweddb
maxDataSize = 10000

maxHotBuckets = 10

splunk_app_windows_infrastructure
[ad-msad]
homePath = $SPLUNK_DB/ad-msad/db
coldPath = $SPLUNK_DB/ad-msad/colddb
thawedPath = $SPLUNK_DB/ad-msad/thaweddb
maxDataSize = 10000
maxHotBuckets = 10

[ad-perfmon]
homePath = $SPLUNK_DB/ad-perfmon/db
coldPath = $SPLUNK_DB/ad-perfmon/colddb
thawedPath = $SPLUNK_DB/ad-perfmon/thaweddb
maxDataSize = 10000
maxHotBuckets = 10

[ad-winevents]
homePath = $SPLUNK_DB/ad-winevents/db
coldPath = $SPLUNK_DB/ad-winevents/colddb
thawedPath = $SPLUNK_DB/ad-winevents/thaweddb
maxDataSize = 10000

maxHotBuckets = 10

Splunk_TA_windows
[win-windows]
homePath = $SPLUNK_DB/win-windows/db
coldPath = $SPLUNK_DB/win-windows/colddb
thawedPath = $SPLUNK_DB/win-windows/thaweddb

[win-wineventlog]
homePath = $SPLUNK_DB/win-wineventlog/db
coldPath = $SPLUNK_DB/win-wineventlog/colddb
thawedPath = $SPLUNK_DB/win-wineventlog/thaweddb

[win-perfmon]
homePath = $SPLUNK_DB/win-perfmon/db
coldPath = $SPLUNK_DB/win-perfmon/colddb
thawedPath = $SPLUNK_DB/win-perfmon/thaweddb

On SplunkUF am keep getting these errors

01-15-2016 03:57:30.303 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonAppDoMonitoring: Error 0x6 occurred during execution
01-15-2016 03:58:30.894 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonStopDriver - Service 'splknetdrv' could not be stopped! Error = 1062
01-15-2016 03:58:30.894 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonStartDriver - StartService failure for splknetdrv! Error = 6. Please check that Windows patch kb 2685811 is installed.
01-15-2016 03:58:30.894 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonAppDoMonitoring: Failed to open monitor device: 0x6
01-15-2016 03:58:30.894 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonAppDoMonitoring: Error 0x6 occurred during execution
01-15-2016 03:59:30.486 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonStopDriver - Service 'splknetdrv' could not be stopped! Error = 1062
01-15-2016 03:59:30.486 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonStartDriver - StartService failure for splknetdrv! Error = 6. Please check that Windows patch kb 2685811 is installed.
01-15-2016 03:59:30.502 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonAppDoMonitoring: Failed to open monitor device: 0x6
01-15-2016 03:59:30.502 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonAppDoMonitoring: Error 0x6 occurred during execution
01-15-2016 04:00:31.903 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonStopDriver - Service 'splknetdrv' could not be stopped! Error = 1062
01-15-2016 04:00:31.903 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonStartDriver - StartService failure for splknetdrv! Error = 6. Please check that Windows patch kb 2685811 is installed.
01-15-2016 04:00:31.903 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonAppDoMonitoring: Failed to open monitor device: 0x6
01-15-2016 04:00:31.903 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonAppDoMonitoring: Error 0x6 occurred during execution
01-15-2016 04:01:31.152 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonStopDriver - Service 'splknetdrv' could not be stopped! Error = 1062
01-15-2016 04:01:31.152 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonStartDriver - StartService failure for splknetdrv! Error = 6. Please check that Windows patch kb 2685811 is installed.
01-15-2016 04:01:31.152 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonAppDoMonitoring: Failed to open monitor device: 0x6
01-15-2016 04:01:31.152 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"" splunk-netmon - NetmonAppDoMonitoring: Error 0x6 occurred during execution