Deployment Architecture

Why are the bucket numbers in colddb out of sequence?

yuelu
Explorer

I see in the db of one of my indexers:
drwx--x--- 3 root root 4096 Aug 25 22:29 db_1503779100_1503700882_4044
drwx--x--- 3 root root 4096 Aug 26 05:05 db_1503802800_1503721750_4045
drwx--x--- 3 root root 4096 Aug 26 11:41 db_1503826800_1503745500_4046
drwx--x--- 3 root root 4096 Aug 26 18:15 db_1503850200_1503769261_4047
......

In the colddb:
......
drwx--x--- 3 root root 4096 Aug 25 00:07 db_1503698700_1503621671_4040
drwx--x--- 3 root root 4096 Aug 25 06:06 db_1503720300_1503641264_4041
drwx--x--- 3 root root 4096 Aug 25 11:35 db_1503739860_1503662781_4042
drwx--x--- 3 root root 4096 Aug 25 16:41 db_1503758400_1503682500_4043
drwx--x--- 3 root root 4096 Sep 11 13:56 db_1490323691_1486237836_3906
drwx--x--- 3 root root 4096 Sep 11 14:44 db_1490325851_1490323116_4115
drwx--x--- 3 root root 4096 Sep 11 15:52 db_1490330171_1490325850_4117
drwx--x--- 3 root root 4096 Sep 21 17:49 db_1491201351_1490329944_4119
drwx--x--- 3 root root 4096 Nov 9 09:13 db_1495411094_1491201516_4154

What can cause the rotation from warm bucket to cold bucket to have a big gap (Sep. 21 - Nov. 9) and the bucket rolling date and bucket number out of order. Before db_1490323691_1486237836_3906, when listing by timestamp, all bucket numbers are in sequence. Should I be concerned?

Thanks.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @yuelu,

Let me explain why there is sequence number out of order, by default splunk will create upto 3 hot buckets and these hot buckets can roll from hot to warm based on some of the parameter (maxHotSpanSecs and maxDataSize, whichever hit first). If we consider bucket db_1490323691_1486237836_3906 in this case earliest event is from 4th Feb 2017 19:50 GMT to latest event is 24th March 2017 02:48 GMT and if I assume that you have default maxHotSpanSecs which is 90 days then you didn't hit this parameter and I assume you didn't hit maxDataSize as well then in this case splunk will roll this bucket when you will have more than 3 hot buckets or when you will restart splunk.

To roll bucket from Warm to Cold is depend on maxWarmDBCount parameter. Here if we consider about bucket db_1495411094_1491201516_4154 which has earliest event from 3rd April 2017 06:38 to latest event 21st May 2017 23:58 GMT but why this bucket has id 4154, this occurs when splunk is not able to parse timestamp properly or your event contains old timestamp so splunk generate new bucket with new id in sequence but contains earliest time and latest time based on events which is present in that bucket so in this case you need to check whether forwarder itself sending events with old timestamp or splunk is not able to parse timestamp properly.

If you still require more info then please provide your indexes.conf configuration so someone from community can help you.

I hope this helps.

Thanks,
Harshil

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...