rebuild statistics to summary index due to network disruption

[wanling]

I encountered an issue in our splunk environment. The network connection between the forwarders and splunk indexer was down for the last few days. The summary index ran as scheduled yet summarised no data. Now the network is back to operation. I thought the fill_summary_index command would help to rebuild the statistics into the summary index for the last few days. Yet, the command returns that there is no search to run.

Question: How can i get the statistical records back to the summary index so that my report wont show the data gaps along the timeline?

[dwaddle]

I come to you from the future!

All joking aside, issues such as these were places where Splunk themselves realized the limitations in summary indexing. It is a great tool for some things, but has some serious architectural gaps.

In the modern world - as of Splunk 6.0 or so - we use Data Models and Data Model acceleration in order to achieve the performance advantages of summary indexing without all of the operational downsides of summary indexing. If you are still using Splunk (this is a 4 year old unanswered post afterall) - then the docs on DMs and Acceleration will give you a hint how to proceed:

http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Aboutdatamodels
http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Acceleratedatamodels

[MickSheppard]

The fill_summary_index command won't see any gaps because it ran, albeit with no data to summarise. This means that there are entries in the summary index for the time period when the data wasn't being forwarded correctly.

In order not to confuse things you will need to remove the errant summary index entries using the delete command. Once you've done that fill_summary_index will generate the results you are looking for.

[MickSheppard]

If you do a search with the correct time specified then pipe the results to delete it will just delete those entries. In order to do this you need to have the delete privilege assigned to your role.

E.g.

index=mysummaryidx earliest=-2d@d latest=-1d@d | delete

[wanling]

you don't mean deleting the whole index, right? can you elaborate how to delete only errant summary index entries? Thanks