Splunk summary data missing

[Myelin]

We're missing all of Splunk's summary data (index=summary_forwarders/summary_indexers, etc). It was working previously and has since stopped- so we know when it broke, but the only changes on that date were networking changes and all other parts of Splunk seem to be working fine (including other indexes). When I look on one of the indexers at the hot buckets for (for example) summary_forwarders I see two old warm buckets and a new one is never created.

We've tried a lot of the basics- restart Splunk, restart the boxes, but there is no change, and have been looking around for clues at all kinds of confs..

Any thoughts on what the problem could be or where to look? We've been banging our heads against the desk for a week on this and it's starting to hurt! I've checked the splunkd log and see where our last hot bucket gets rolled to warm, but never see any entry or error about trying to create the next bucket in sequence.

[Myelin · Accepted solution]

We finally solved this issue. Even though we restarted Splunk through the CLI and the entire box itself- this had no effect. I went into the WebUI -> Manager -> Indexes. In here I disabled the summary_forwarders index and restarted Splunk as it instructed. I then enabled the summary_forwarders index. After this ALL of the summary_* indexes are now populated and back filling all the past data.

[Myelin]

Checked the DM app. Nothing is Disabled there and the schedules look normal. I should have noted this is 4.3.2 (we are working on an upgrade!).

When I look at the job history I see all kinds of searches running successfully in the DM app, including "All forwarders - regenerator summary index" which seems to have all the data we're looking for, and references index=summary_forwarders. So it looks like the searches are actually working fine as well...

But once again searching: index="summary_forwarders" returns nothing.

[jbrodsky_splunk]

This appears related to the Deployment Monitor app. And the older version, that used summary indexes. Summary indexes are normally populated via scheduled searches - every 30m I think. Are those searches still running on the schedule that you expect them to? Perhaps they're disabled? Perhaps they are failing for some other reason? Perhaps someone deleted them? Look at the config of the DM app, and saved search history.

http://answers.splunk.com/answers/34532/deployment-monitor-issue-no-data-in-summary-indexes
http://answers.splunk.com/answers/48883/deployment-monitor-summary-indexes-issue