Deployment Architecture

What is the typical size of the (compressed) buckets; given 10gb a day in indexed data, what kind of growth will I see on disk?

Dimitri_McKay
Splunk Employee
Splunk Employee

I know this is dependent on the variance of the data being indexed, but since the indexing mechanism is proprietary, I’d like some real-world numbers.

0 Karma
1 Solution

Dimitri_McKay
Splunk Employee
Splunk Employee

50% (2:1 ratio) is a safe bet. That includes not only the indexes but also the compressed raw log data. So allocating 5GB per day is a good bet, though I'd probably add at least 20% on top for growth.

Also when talking storage, you'll want to consider average search time. So, the majority of searches which take place are usually "last 24 hours" or "last 7 days" but rarely do most searches go beyond that 7 day period. So having 40GB of local storage (as fast as possible as that disk is going to handle collection, compression, indexing and search for that short time period). Then it can be pushed out to slower storage afterward.

View solution in original post

Dimitri_McKay
Splunk Employee
Splunk Employee

50% (2:1 ratio) is a safe bet. That includes not only the indexes but also the compressed raw log data. So allocating 5GB per day is a good bet, though I'd probably add at least 20% on top for growth.

Also when talking storage, you'll want to consider average search time. So, the majority of searches which take place are usually "last 24 hours" or "last 7 days" but rarely do most searches go beyond that 7 day period. So having 40GB of local storage (as fast as possible as that disk is going to handle collection, compression, indexing and search for that short time period). Then it can be pushed out to slower storage afterward.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...