Deployment Architecture

Bucket number exceeds maxHotBuckets value

osec2a
New Member

Hi,

In my splunk configurtation I have defined the maxHotBuckets to default value, so 3.
When I monitor my indexers I see that the number of Hot buckets exceeds the value 3.
Can anyone explain me why the number of hot buckets can exceed the maximum number of hot buckets?

Thanks

0 Karma

osec2a
New Member

@mmodestino This is my configuration in moniroting console

alt text

When I use dbinspect command to display each nomber of bucket per indexer this is what I get :

alt text

How can indexer have more than 3 (maxHotBuckets) hot buckets?

Thanks

0 Karma

mattymo
Splunk Employee
Splunk Employee

i recommend validating from the CLI. there is not enough info from dbinspect in the search above and I am not clear on whether you are even running a cluster?? I see 1 hot per indexer in that screenshot...Jump on the cli of that indexer and confirm what you are seeing in $SPLUNK_HOME/var/lib/splunk/<yourIndex>/db and look for the naming convention kellewic alluded to above. (hot_ and ####_GUID)

example from my standalone instance

[root@n00bserver db]# pwd
/home/splunker/splunk/var/lib/splunk/n00blab/db
[root@n00bserver db]# ls -la | grep hot
drwx--x---   3 splunker splunker  4096 Aug  9 15:02 hot_v1_342
drwx--x---   3 splunker splunker  4096 Aug  9 15:02 hot_v1_343
- MattyMo
0 Karma

kellewic
Path Finder

dbinspect is counting the replicated hot buckets.

Look on one of your indexers for that index; you should see 3x buckets like "hot_" and N more like "####_GUID". Those can even exceed what dbinspect says but the ones with more than just "rawdata" is what's being counted in addition to the originating hot buckets.

Or if you have multiple pipelines, this can happen as well as inventsekar pointed out.

Easy check on the replicated front; try:

|dbinspect index=INDEX
|where state="hot"
|eval replicated=if(match(path, "/rb_"), "Y", if(state="hot" AND match(path, "/\d+_[-A-F0-9]+"), "Y", "N"))
|stats count by splunk_server, replicated
|sort replicated

Those with "N" will equal 3; those with "Y" will be the remainder.

Can also check:

|rest /services/data/indexes-extended
|where title="INDEX"
|table splunk_server, maxHotBuckets, bucket_dirs.home.hot_bucket_count
0 Karma

inventsekar
Ultra Champion
  • Note that this limit will be applied per ingestion pipeline. For more information about multiple ingestion pipelines see parallelIngestionPipelines in server.conf.spec file.
  • With N parallel ingestion pipelines the maximum number of hot buckets across all of the ingestion pipelines will be N * maxHotBuckets but maxHotBuckets for each ingestion pipeline. Each ingestion pipeline will independently write to and manage up to maxHotBuckets number of hot buckets. As a consequence of this, when multiple ingestion pipelines are used, there may be multiple (dependent on number of ingestion pipelines configured) hot buckets with events with overlapping time ranges.

https://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/Indexesconf

0 Karma

osec2a
New Member

In server.conf parallelIngestionPipelines value is 1, so the maximum number of hot bucket is always 3.

0 Karma

kellewic
Path Finder

How are you monitoring them? Is it counting replicated hot buckets possibly?

0 Karma

osec2a
New Member

I have a distributed envrironment, so I manage all my indexers on a deploployer server
I don't think it is counting replicated hot buckets.
When I use dbinspect command, I see the same result

0 Karma

mattymo
Splunk Employee
Splunk Employee

can you be more specific and provide an example of what you are seeing?

so you are not clustering? And you are seeing more than 3 hot buckets per index? How many do you see??

- MattyMo
0 Karma

osec2a
New Member

I add an answer beacause I can't add img in comments

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...