Monitoring Splunk

Deployment Monitor Issue - no data in summary indexes

apringle
Explorer

I just added a new Universal Forwarder to our Splunk deployment (we previously were running everything on a single server, this is the first attempt at a Forwarder/Receiver). For the most part, everything seems to be working okay. We're receiving data on the indexer, able to search it, etc.

Then I enabled the Deployment Monitor app, but it not showing any data. It seems that our summary* indexes are empty (if I run a search with index=summary or index=summary_indexers, I get no results)

I do see jobs running in the Searches & Reports management interface, and I've also tried to backfill the data inside of Deployment Monitor, with no luck.

I see the following log entries in splunkd.log regarding the summary indexes. This repeats for all of the summary indexes (summary, summary_fowarders, summary_hosts, summary_pools, summary_sources, summary_sourcetypes).

11-16-2011 16:15:09.484 -0700 INFO  IndexProcessor - Initializing index: summary
11-16-2011 16:15:09.484 -0700 INFO  HotDBManager - setting hot mgr params: /opt/splunk/var/lib/splunk/summarydb/db maxHotSpanSecs=7776000 maxHotBuckets=3 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
11-16-2011 16:15:09.484 -0700 INFO  databasePartitionPolicy - index summary initialized with [300,60,188697600,,,,786432000,20,true,500000,5,5,false,3,0,_blocksignature,7776000,1000000,0,3,77760000,2592000,131072,25,0,15,0,0,-1,18446744073709551615ms]
11-16-2011 16:15:09.484 -0700 INFO  databasePartitionPolicy - openDatabase for /opt/splunk/var/lib/splunk/summarydb/db
11-16-2011 16:15:09.484 -0700 INFO  databasePartitionPolicy - We are running on a pre-existing database opening ...
11-16-2011 16:15:09.484 -0700 INFO  databasePartitionPolicy - No databases found starting fresh !
11-16-2011 16:15:09.484 -0700 INFO  databasePartitionPolicy - CREATION TIME for /opt/splunk/var/lib/splunk/summarydb/db : 1321481049
11-16-2011 16:15:09.484 -0700 WARN  databasePartitionPolicy - failed to open metadata for /opt/splunk/var/lib/splunk/summarydb/db, will attempt full rebuild
11-16-2011 16:15:09.485 -0700 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/opt/splunk/var/lib/splunk/summarydb/db reason=initopenMetaData failed
11-16-2011 16:15:09.485 -0700 INFO  databasePartitionPolicy - clearing existing internal aggregate metadata (/opt/splunk/var/lib/splunk/summarydb/db)
11-16-2011 16:15:09.485 -0700 INFO  databasePartitionPolicy - currentId for /opt/splunk/var/lib/splunk/summarydb/db after openDatabases = 0
1 Solution

apringle
Explorer

Found the issue. Our system/local/inputs.conf file on our indexer, for some reason, had this:

[monitor://$SPLUNK_HOME/var/log/splunk]
disabled = 1

I'm not sure why this was there, probably some relic of the past, but re-enabling this monitor caused everything to start working with the deployment monitor.

View solution in original post

araitz
Splunk Employee
Splunk Employee

Very weird! Glad you were able to find the solution.

0 Karma

apringle
Explorer

Thank you for the suggestions - this search returned nothing, which caused me to dig into this and find the solution.

0 Karma

apringle
Explorer

Found the issue. Our system/local/inputs.conf file on our indexer, for some reason, had this:

[monitor://$SPLUNK_HOME/var/log/splunk]
disabled = 1

I'm not sure why this was there, probably some relic of the past, but re-enabling this monitor caused everything to start working with the deployment monitor.

RiccardoV
Communicator

@apringle after 18 months, I LOVE YOU. I had the same problem and i fix it thanks to your auto-answer 😄

0 Karma

apringle
Explorer

Thanks, I edited my answer to read disabled = 1. I initially pasted in my corrected version by accident.

hexx
Splunk Employee
Splunk Employee

That's a bit odd, "disabled = 0" would indicate that the input was, in fact, enabled - as is expected.

araitz
Splunk Employee
Splunk Employee

What do you see if you search your indexer's internal index for the following:

 index="_internal" source="*metrics.log" group=tcpin_connections"

Specifically, do you see any events from your universal forwarder (i.e. host=your_uf_host_name)?

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...