Could I set the the time limit on Hot/Warm buckets ?

[leo_wang]

 I want to control the data age of my indexes. 
 for example :   1-years data keeps on hot/warm bucket, and 2-years on colddb

 But I can't find the parameter to set the time limits on Warm buckets.
 Both "maxWarmDBCount" , "coldPath.maxDataSizeMB" are related to the data size , but data age.

 Is that possible to control the data age for warm buckets ?

[jkat54]

@leo_wang ... you might want to come look at the newer answers here.

[jkat54]

Technically that is incorrect @coltwanger.

You can control the age of data in your hot buckets rather nicely assuming you have your timestamping correct.

[indexName]
maxHotSpanSecs = 86401
maxHotIdleSecs = 86401

If you make your hot buckets cut off at one day like above then you should easily be able to calculate how many warm buckets make up a year, etc. But that will be a lot of buckets and you should probably go for 1 month per bucket instead.

See my more detailed answer and visual aid here:

https://answers.splunk.com/answers/499760/in-distributed-management-console-dmc-why-is-dataa.html#an...

[coltwanger]

Thank you for the clarification! I'll have to look into these settings for sure. We've got some certainly ridiculous retention we need to follow (8 years total for most systems). However, we try to keep all of our most frequently searched data on our flash storage and run the cold/frozen dirs on some slower cheaper disks. Some data we plan on keeping on flash for the entire 8 years of retention because we end up doing a lot of "needle in the haystack" searches on it.

For other internal groups that we let use Splunk it's generally 30 days in flash (hot/warm), 60 in spinning disks (cold) and then rolled off to frozen. I've had to adjust the db sizes to accommodate for the 90 day total retention on these indexes, but maybe I can use the above settings to alleviate some of my issues 🙂

[jkat54]

OhSnap by itself isn't intrinsically bad though. My understanding is if you're ingesting a small time frame of data (such as just the last 15 days, or if your data only contains 10 days of events), and you set your cold To frozen secs to something low like 30 days... Then surely a cluster can handle '30 buckets x SF x RF.' However if you're ingesting something like a windows application log that has 3 years of history in it, and you leave default frozen time period in secs, then you could create a bucket storm. The storm would be exacerbated if it was part of a multisite cluster etc. Do you agree @dwaddle

[dwaddle]

I may be suffering from too much negative feedback,but ohSnap has only ever brought me tears. Last time, it was a cluster that decided to misbehave over Thanksgiving holiday because one app out of hundreds had timestamps being parsed wrong. This led to several new hot buckets per minute, most of which had exactly one event.

I'll concede that IF you know you have well-ordered timestamps and a low retention and a high enough value for maxHotBuckets that ohSnap may not be bad. But every time it and I have crossed paths I have wound up in tears.

[jkat54]

Thanks for the clarification on slack chat. I want to be clear to anyone reading that 86400 or 3600 is a very bad idea. Adding just 1 second to each is the best approach if you must use such low settings for maxHotSpanSecs.

[dwaddle]

For maxHostSpanSecs, Do 86399, do 86401 ... DO NOT DO 86400 (or 3600).

• If set to 3600 (1 hour), or 86400 (1 day), becomes also the lower bound
  of hot bucket timespans. Further, snapping behavior (i.e. ohSnap) is activated, whereby hot bucket boundaries will be set at exactly the hour or day mark, relative to local midnight.

The thing that usually happens when you enable ohSnap (which notice you can't do 86400 without ohSnap coming on implicitly) is that you'll make a large number of small buckets. In a cluster, this can bring the whole cluster to its knees and break things quite terribly.

[jkat54]

I forgot about ohSnap! I've been using this as an example too 😞

[coltwanger · Accepted solution]

You can't set a time retention on hot or warm buckets. Only by size in MB. This is one of my huge gripes in Splunk.

You can use the Monitoring Console to gain some insight into how much data is in your hot/warm directories (x amount of days, for instance). The only time retention I've found is based on moving data from cold into frozen.

[slebbie_splunk]

I downvoted this post because you can set the retention size

[scheng_splunk]

@coltwanger is actually right on this. The maxHotSpanSecs and maxHotIdleSecs won't kick in until the idle time gap between the last most recent event arriving at the hot bucket and the newly arriving event exceeding the value defined. Hence if events are constantly coming into the hot bucket, it won't roll even maxHotSpanSecs/maxHotIdleSecs set to 1hr/1day etc.

maxHotSpanSecs =
* Upper bound of timespan of hot/warm buckets in seconds.
* NOTE: If you set this too small, you can get an explosion of hot/warm
buckets in the filesystem.
* NOTE: If you set maxHotBuckets to 1, Splunk attempts to send all
events to the single hot bucket and maxHotSpanSeconds will not be
enforced.
* If you set this parameter to less than 3600, it will be automatically
reset to 3600.
* This is an advanced parameter that should be set
with care and understanding of the characteristics of your data.
* Highest legal value is 4294967295
* Defaults to 7776000 seconds (90 days).
* Note that this limit will be applied per ingestion pipeline. For more
information about multiple ingestion pipelines see parallelIngestionPipelines
in server.conf.spec file.
* With N parallel ingestion pipelines, each ingestion pipeline will write to
and manage its own set of hot buckets, without taking into account the state
of hot buckets managed by other ingestion pipelines. Each ingestion pipeline
will independently apply this setting only to its own set of hot buckets.
* NOTE: the bucket timespan snapping behavior is removed from this setting.
See the 6.5 spec file for details of this behavior.

maxHotIdleSecs =
* Provides a ceiling for buckets to stay in hot status without receiving any
data.
* If a hot bucket receives no data for more than maxHotIdleSecs seconds,
Splunk rolls it to warm.
* This setting operates independently of maxHotBuckets, which can also cause
hot buckets to roll.
* A value of 0 turns off the idle check (equivalent to infinite idle time).
* Highest legal value is 4294967295

* Defaults to 0.

Feedback from engineering:
"maxHotSpanSecs would be of lesser priority from the default settings within indexes.conf. We'd like to make sure first that we don't make too many small buckets so we'd prefer to "expand" a hot bucket time range rather than be strict with it. "

"There is no rule about rolling to warm when exceeding the maxHotSpanSecs. We roll a bucket to warm because it's been idle too long, we've reached maxHotBuckets (-1 if no quarantine bucket) or a bucket has reached a certain size in disk."

"If we cannot find a bucket with the correct time range for an event, we find the next best bucket and expand that range. To check that we still honor the maxHotSpanSecs without the influence of the above behavior"

[damode]

Hi Scheng,

For how long does the warm bucket retains data before rolling into cold bucket ? is there any attribute to modify that ?

I have requirement of 30 day active and 90 day cold storage.
To achieve this , can I do frozenTimePeriodInSecs = 10368000 (120 days) and maxHotSpanSecs=2592000 (30 days) ?

[jkat54]

I give up... stupid buckets