Hot buckets filling up

[fred_mcghee]

I have 36 indexers each with 2.7gb of space. There are currently 29 of the 36 at capacity and keeping entering abnormal state. How can I get the indexes to roll the data or open up space to solve the alerting?

[richgalloway]

You appear to have at least two problems:
1) Your data is not evenly distributed across your indexers. Even distribution would have kept the 29 drives from filling up quickly and would improve search performance, but is not your main problem.
2) Your indexes are mis-configured. Volumes should be sized so they don't, combined, exceed the available storage. Don't forget to allow for file system overhead, data model accelerations, and replicated buckets. We'd have to know more about your index configuration to offer specific advise.

Also. you may have too many replicated buckets. Consider lowering your replication factor.
Make sure $SPLUNK_DB is not sharing storage with $SPLUNK_HOME, the operating system, or another application.

[fred_mcghee]

Hello Rich

We are set to 2 searchable and 3 replicated right now. I believe we are sized too small. We have 2.7 gb of space on all the indexers and 2.6 is used. I think it was configure to have 30 days of searchable data in HOT and I think that is too much data. Do you think increasing the storage of the indexers is the best option or decrease the days os HOT searchable?

[richgalloway]

Adding more storage is the best idea, but you may find yourself in the same situation later if you don't get your configuration right. Once you have the settings tuned buckets should roll before the storage fills.