Getting Data In

How to configure maxVolumeDataSizeMB when homePath and coldPath are both set to the same volume?

teedilo
Path Finder

$SPLUNK_DB for one of our Splunk Search Servers filled up recently. The root cause was apparently due to significant recent increases in the _audit index on this server due to recent increases in search activity. We have dedicated Indexers as well, but apparently indexing for the _audit and _internal indexes (and possibly other internal indexes) still occurs on the Search Servers. We had been getting by with the default indexes.conf file (/opt/splunk/etc/system/default/indexes.conf) on the Search Server, but apparently that will no longer be possible.

We recently changed the indexes.conf files on our Indexers to use the volume based parameters ([volume:home], [volume:cold], etc.). That has worked well for the Indexers, where the hot/warm volume is a distinct volume and file system from the cold volume. For our Search servers, however, we just have all hot/warm and cold buckets in $SPLUNK_DB (/opt/splunk/var/lib/splunk). That being said, I would still like to use the volume based parameters in a new (non-default) indexes.conf file for the Search Servers, since I think it provides the best "set-it-and-forget-it" control, letting Splunk manage the size of all indexes based upon the volume size. Finally, to my question...

What's not clear to me is how to specify the parameters in [volume:home] and [volume:cold] stanzas in this scenario. Specifically, do I specify exactly the same maxVolumeDataSizeMB parameter value for each stanza? For example, if the size of the $SPLUNK_DB volume is 14 GB (14336 MB), is this how I would specify these parameters, or is this saying that I have 14 GB of hot/warm storage, and another 14 GB of cold storage?

[volume:home]
path = $SPLUNK_DB
maxVolumeDataSizeMB = 14336

[volume:cold]
path = $SPLUNK_DB
maxVolumeDataSizeMB = 14336

[_audit]
coldPath = volume:cold/audit/colddb
homePath = volume:home/audit/db
thawedPath = $SPLUNK_DB/audit/thaweddb

[_internal]
coldPath = volume:cold/_internaldb/colddb
homePath = volume:home/_internaldb/db
thawedPath = $SPLUNK_DB/_internaldb/thaweddb

This article suggests that it's possible to use volume based parameters with a single volume for hot/warm and cold buckets:


When a volume containing warm buckets reaches its maxVolumeDataSizeMB, it starts rolling buckets to cold. When a volume containing cold buckets reaches its maxVolumeDataSizeMB, it starts rolling buckets to frozen. If a volume contains both warm and cold buckets (which will happen if an index's homePath and coldPath are both set to the same volume), the oldest bucket will be rolled to frozen.

But the article does not include an example of how to specify these parameters in this scenario. It's also not clear to me from the indexes.conf documentation.

I should add that 14 GB for $SPLUNK_DB (/opt/splunk/var/lib/splunk) is an approximation. The size of $SPLUNK_HOME (/opt/splunk) is actually 19.6 GB. I just want to say 14 GB is available for $SPLUNK_DB to account for the other stuff under /opt/splunk (e.g., the Splunk installation, configuration files, and some overhead for the minimum free space that Splunk requires in order to work). I wish it was possible to provide tighter control over the index space in this situation, but I don't think that is possible without putting $SPLUNK_DB in a separate dedicated file system by itself.

0 Karma
1 Solution

teedilo
Path Finder

I figured out the answer to this on my own. Even though the paths for the hot/warm and cold volumes are the same in this situation, you need to specify maxVolumeDataSizeMB values for each "volume" that partitions out how much space you want for hot/warm vs. cold buckets.

In my case, I didn't want any cold buckets at all. I learned that Splunk isn't designed to just "skip" cold and go directly to frozen, so the best I could do was to specify the lowest allowable number for maxVolumeDataSizeMB for cold, which is 1. That way buckets that are moved to cold will likely be rolled to frozen (deleted in this case since no archive) immediately, or nearly so.

For the hot/warm volume, I specified a maxVolumeDataSizeMB that allowed for 5 GB of additional stuff under /opt/splunk (since the data is under /opt/splunk/var/lib/splunk).

The end result was the settings shown below.

[default]
rotatePeriodInSecs = 10
maxWarmDBCount = 4294967295
maxTotalDataSizeMB = 4294967295
maxHotBuckets = 5
maxDataSize = 1024

[volume:home]
path = $SPLUNK_DB
maxVolumeDataSizeMB = 15053

[volume:cold]
path = $SPLUNK_DB
maxVolumeDataSizeMB = 1

[_audit]
homePath   = volume:home/audit/db
coldPath   = volume:cold/audit/colddb
thawedPath = /opt/splunk/var/lib/splunk/audit/thaweddb

[_internal]
homePath   = volume:home/_internaldb/db
coldPath   = volume:cold/_internaldb/colddb
thawedPath = /opt/splunk/var/lib/splunk/_internaldb/thaweddb

This solution was not intuitive, so using the volume size configuration settings in the single volume scenario should ideally be included in an example in the Splunk documentation.

As I had mentioned in my original post, the servers affected by this are our Search servers, as they have a single volume for Splunk data, whereas our Indexers have separate hot/warm and cold volumes. I later found this article, which talks about forwarding search head data to the Indexers as a best practice. I might implement that at a later time, at which point I could undo the indexes.conf settings that I described above. The settings required to do this seem straightforward enough, but the article does not address migrating existing data from the Search servers to the Indexers, so I'll need to figure that out before attempting this.

View solution in original post

0 Karma

yannK
Splunk Employee
Splunk Employee
0 Karma

teedilo
Path Finder

I figured out the answer to this on my own. Even though the paths for the hot/warm and cold volumes are the same in this situation, you need to specify maxVolumeDataSizeMB values for each "volume" that partitions out how much space you want for hot/warm vs. cold buckets.

In my case, I didn't want any cold buckets at all. I learned that Splunk isn't designed to just "skip" cold and go directly to frozen, so the best I could do was to specify the lowest allowable number for maxVolumeDataSizeMB for cold, which is 1. That way buckets that are moved to cold will likely be rolled to frozen (deleted in this case since no archive) immediately, or nearly so.

For the hot/warm volume, I specified a maxVolumeDataSizeMB that allowed for 5 GB of additional stuff under /opt/splunk (since the data is under /opt/splunk/var/lib/splunk).

The end result was the settings shown below.

[default]
rotatePeriodInSecs = 10
maxWarmDBCount = 4294967295
maxTotalDataSizeMB = 4294967295
maxHotBuckets = 5
maxDataSize = 1024

[volume:home]
path = $SPLUNK_DB
maxVolumeDataSizeMB = 15053

[volume:cold]
path = $SPLUNK_DB
maxVolumeDataSizeMB = 1

[_audit]
homePath   = volume:home/audit/db
coldPath   = volume:cold/audit/colddb
thawedPath = /opt/splunk/var/lib/splunk/audit/thaweddb

[_internal]
homePath   = volume:home/_internaldb/db
coldPath   = volume:cold/_internaldb/colddb
thawedPath = /opt/splunk/var/lib/splunk/_internaldb/thaweddb

This solution was not intuitive, so using the volume size configuration settings in the single volume scenario should ideally be included in an example in the Splunk documentation.

As I had mentioned in my original post, the servers affected by this are our Search servers, as they have a single volume for Splunk data, whereas our Indexers have separate hot/warm and cold volumes. I later found this article, which talks about forwarding search head data to the Indexers as a best practice. I might implement that at a later time, at which point I could undo the indexes.conf settings that I described above. The settings required to do this seem straightforward enough, but the article does not address migrating existing data from the Search servers to the Indexers, so I'll need to figure that out before attempting this.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...