I'm trying to understand the compression numbers provided by Splunk. Given a compression of, say, 40%, on a volume of 100 GB. What does that translate to on disk for storage purposes?
Is it 60 GB (100 GB x (100-40)) OR
is it 40% x 100 GB = 40 GB?
Or, something else?
Not clear what number's you're referring to but compression ratio should always be calculated as:
Compression Ratio = (Uncompressed Size)/(Compressed Size)
Also, compression ratio expressed in percent
does not make much sense. Storage savings on the other hand are a different story.
Example:
Uncompressed = 100GB, Compressed = 40GB
compression ratio = 100/40 = 2.5
OR alternatively noted as 2.5:1
savings % = 100 * (1 - 40GB/100GB) = 60%
Splunk Document Link Changed. Follow below Link for your reference :
https://docs.splunk.com/Documentation/Splunk/latest/Capacity/Estimateyourstoragerequirements
This docs topic talks about calculating storage:
http://docs.splunk.com/Documentation/Splunk/latest/Installation/Estimateyourstoragerequirements
Also, is this a reliable number to use for storage calculation? What I mean is, does this diskTotalinMB include all associated files that require space for that index? Is this the "du" for the entire index (hot and cold) and all files within?
Not clear what number's you're referring to but compression ratio should always be calculated as:
Compression Ratio = (Uncompressed Size)/(Compressed Size)
Also, compression ratio expressed in percent
does not make much sense. Storage savings on the other hand are a different story.
Example:
Uncompressed = 100GB, Compressed = 40GB
compression ratio = 100/40 = 2.5
OR alternatively noted as 2.5:1
savings % = 100 * (1 - 40GB/100GB) = 60%
That's why this is confusing -- the wording is wrong.
...char limit
...A more exact statement would be your former one: "100GB of raw data indexed takes up 40% of its original volume"
No. You're confusing the compressed size of something expressed as a percentage of the original with compression ratio of a certain mechanism. Mathematically and technically speaking, the compression ratio is never expressed or noted in percent. Instead notations such as x/y
, x:y
or alike are used.
"100GB indexed at 40% compression rate = 40GB on disk." This is a wrongly worded statement about percentages and rates. That's like saying: a $100 pair of jeans sold at 25% discount will cost $25.
Compression expressed as percent absolutely makes sense! "The data takes up X% of its original volume." 100GB indexed at 40% compression rate = 40GB on disk.
diskTotalinMB = rawTotalinMB * (compression * 100)
I checked the stats and the numbers work out to your answer.