Monitoring Splunk

Trying to understand compression - given % compression of X volume data, how much on disk is required?

the_wolverine
Champion

I'm trying to understand the compression numbers provided by Splunk. Given a compression of, say, 40%, on a volume of 100 GB. What does that translate to on disk for storage purposes?

Is it 60 GB (100 GB x (100-40)) OR
is it 40% x 100 GB = 40 GB?
Or, something else?

alt text

0 Karma
1 Solution

_d_
Splunk Employee
Splunk Employee

Not clear what number's you're referring to but compression ratio should always be calculated as:

Compression Ratio = (Uncompressed Size)/(Compressed Size)

Also, compression ratio expressed in percent does not make much sense. Storage savings on the other hand are a different story.

Example:

Uncompressed = 100GB, Compressed = 40GB

compression ratio = 100/40 = 2.5 OR alternatively noted as 2.5:1

savings % = 100 * (1 - 40GB/100GB) = 60%

View solution in original post

hakeniz
Loves-to-Learn Lots

Splunk Document Link Changed. Follow below Link for your reference : 

https://docs.splunk.com/Documentation/Splunk/latest/Capacity/Estimateyourstoragerequirements

0 Karma

jlaw
Splunk Employee
Splunk Employee
0 Karma

the_wolverine
Champion

Also, is this a reliable number to use for storage calculation? What I mean is, does this diskTotalinMB include all associated files that require space for that index? Is this the "du" for the entire index (hot and cold) and all files within?

0 Karma

_d_
Splunk Employee
Splunk Employee

Not clear what number's you're referring to but compression ratio should always be calculated as:

Compression Ratio = (Uncompressed Size)/(Compressed Size)

Also, compression ratio expressed in percent does not make much sense. Storage savings on the other hand are a different story.

Example:

Uncompressed = 100GB, Compressed = 40GB

compression ratio = 100/40 = 2.5 OR alternatively noted as 2.5:1

savings % = 100 * (1 - 40GB/100GB) = 60%

the_wolverine
Champion

That's why this is confusing -- the wording is wrong.

0 Karma

_d_
Splunk Employee
Splunk Employee

...char limit...A more exact statement would be your former one: "100GB of raw data indexed takes up 40% of its original volume"

0 Karma

_d_
Splunk Employee
Splunk Employee

No. You're confusing the compressed size of something expressed as a percentage of the original with compression ratio of a certain mechanism. Mathematically and technically speaking, the compression ratio is never expressed or noted in percent. Instead notations such as x/y, x:y or alike are used.
"100GB indexed at 40% compression rate = 40GB on disk." This is a wrongly worded statement about percentages and rates. That's like saying: a $100 pair of jeans sold at 25% discount will cost $25.

sowings
Splunk Employee
Splunk Employee

Compression expressed as percent absolutely makes sense! "The data takes up X% of its original volume." 100GB indexed at 40% compression rate = 40GB on disk.

0 Karma

the_wolverine
Champion

diskTotalinMB = rawTotalinMB * (compression * 100)

0 Karma

the_wolverine
Champion

I checked the stats and the numbers work out to your answer.

0 Karma
Get Updates on the Splunk Community!

More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk

Tuesday, May 14, 2024  |  11AM PT / 2PM ET Register to Attend Join us for this Tech Talk and learn how to ...

.conf24 | Personalize your .conf experience with Learning Paths!

Personalize your .conf24 Experience Learning paths allow you to level up your skill sets and dive deeper ...

Threat Hunting Unlocked: How to Uplevel Your Threat Hunting With the PEAK Framework ...

WATCH NOWAs AI starts tackling low level alerts, it's more critical than ever to uplevel your threat hunting ...