Reason:
The above health warning message is shown when the number of hot buckets created to index the data in particular hour/day crosses or reaches the threshold limit defined in system/health.conf.
Reason behind the creation of too many hot buckets:
Splunk uses buckets as an index directory to index the data. Now, when an event comes to Splunk for indexing, the new hot bucket will be created for that event or the event is indexed to the existing one of the hot buckets as per the event's timestamp and constraints on the index(refer to maxHotBuckets, maxHotSpanSecs and quarantinePastSecs parameters of indexes.conf).
So, if the event is having the older/strange timestamp than acceptable for the existing buckets, then Splunk will create a new hot bucket to index that event. Hence, the number of hot buckets created may reach or cross the threshold.
When DATETIME_CONFIG parameter of props.conf is not set explicitly, Splunk will try to find the timestamp in an event itself. So, While extracting the timestamp from the events, if Splunk finds the value that matches the valid timestamp format, it would parse the same value as event time and would index it using that timestamp only, which may cause Splunk to create new hot bucket as the timestamp is not properly extracted.
Consider adding timestamp attributes in propps.conf to make Splunk extract timestamp properly.
Link: Configure timestamp recognition
... View more