Yes, the issues are still resolved, for months now. I haven't seen the warning again.
The main cause of this issue is most likely going to be because the timestamps on the data you are feeding in are all over the place. Splunk wants to be mostly chronological, so the buckets contain data from a certain window of time.
For example, if your max buckets was set to say 3, and they will hold data from 6 hour windows, and you are feeding in data which is within an 18 hour window, things will be fine, as it can put it in the appropriate 6 hour bucket. If you then give it some data with a random time from outside that 18 hour window, it doesn't have a bucket to put it in.
This will cause it to force close/roll one of the hot buckets early, and create a new bucket for the "random" data. If you then resume giving it data from within the 6 hour window of the hot bucket which was just force rolled to warm in order to create the hot bucket for the random data, now you're back to the same issue; Splunk doesn't have a bucket to put it in.
This will cause a repeat of the "roll hot to warm; create new bucket" process, on one of the other buckets, causing it to roll early. Warm buckets cannot be rolled back to hot buckets, Splunk only creates new ones, so if you keep feeding it data with timestamps all over the place, outside of the window of time it has buckets for, it's going to cause buckets to have to be rolled early and new, overlapping ones created constantly, which is what causes this issue.
As I said above, the 2 things which fixed it for me was to perform searches of data "from the future", which identified data sources that were in GMT/UTC vs GMT-5/-6/-7 (where my sources actually are), and to fix the timezone of the timestamps at the source, (or by making sure props.conf contains parser definitions for your specific data sources in order to mangle them to the correct timezone), and also by increasing the maximum hot buckets Splunk is allowed to juggle at once, in my case from 3 to 5.
Just increasing the max buckets may "fix" the issue, but now you're potentially going to end up with records that are incorrectly timestamped all over the place, and your queries won't even find/return them, since they will be in buckets which don't get searched as they are outside the time frame your query is looking in.
... View more
