I just added a new Universal Forwarder to our Splunk deployment (we previously were running everything on a single server, this is the first attempt at a Forwarder/Receiver). For the most part, everything seems to be working okay. We're receiving data on the indexer, able to search it, etc.
Then I enabled the Deployment Monitor app, but it not showing any data. It seems that our summary* indexes are empty (if I run a search with index=summary or index=summary_indexers, I get no results)
I do see jobs running in the Searches & Reports management interface, and I've also tried to backfill the data inside of Deployment Monitor, with no luck.
I see the following log entries in splunkd.log regarding the summary indexes. This repeats for all of the summary indexes (summary, summary_fowarders, summary_hosts, summary_pools, summary_sources, summary_sourcetypes).
11-16-2011 16:15:09.484 -0700 INFO IndexProcessor - Initializing index: summary
11-16-2011 16:15:09.484 -0700 INFO HotDBManager - setting hot mgr params: /opt/splunk/var/lib/splunk/summarydb/db maxHotSpanSecs=7776000 maxHotBuckets=3 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
11-16-2011 16:15:09.484 -0700 INFO databasePartitionPolicy - index summary initialized with [300,60,188697600,,,,786432000,20,true,500000,5,5,false,3,0,_blocksignature,7776000,1000000,0,3,77760000,2592000,131072,25,0,15,0,0,-1,18446744073709551615ms]
11-16-2011 16:15:09.484 -0700 INFO databasePartitionPolicy - openDatabase for /opt/splunk/var/lib/splunk/summarydb/db
11-16-2011 16:15:09.484 -0700 INFO databasePartitionPolicy - We are running on a pre-existing database opening ...
11-16-2011 16:15:09.484 -0700 INFO databasePartitionPolicy - No databases found starting fresh !
11-16-2011 16:15:09.484 -0700 INFO databasePartitionPolicy - CREATION TIME for /opt/splunk/var/lib/splunk/summarydb/db : 1321481049
11-16-2011 16:15:09.484 -0700 WARN databasePartitionPolicy - failed to open metadata for /opt/splunk/var/lib/splunk/summarydb/db, will attempt full rebuild
11-16-2011 16:15:09.485 -0700 INFO databasePartitionPolicy - rebuildMetadata called: full=true path=/opt/splunk/var/lib/splunk/summarydb/db reason=initopenMetaData failed
11-16-2011 16:15:09.485 -0700 INFO databasePartitionPolicy - clearing existing internal aggregate metadata (/opt/splunk/var/lib/splunk/summarydb/db)
11-16-2011 16:15:09.485 -0700 INFO databasePartitionPolicy - currentId for /opt/splunk/var/lib/splunk/summarydb/db after openDatabases = 0
... View more