Splunk Dev

Min IOPS Requirement for 1 TB indexing /per day

Prakash493
Communicator

Hi , Where can i find the Min IOPS requirement for the data ingestion of 1 TB/DAY with 12 indexers ?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Prakash493,
Splunk recommends at least 800 IOPS, but all you can have it's obviously better!
for more infos see at https://docs.splunk.com/Documentation/Splunk/8.0.0/Capacity/Referencehardware

Ciao.
Giuseppe

0 Karma

Prakash493
Communicator

Yes but it doesn't say about the data ingestion per day. It might be 1 GB/DAY or 100 GB/DAY , So i am looking to best iops for 1 TB/DAY ingestion requirement.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Prakash493,
as I said Splunk requires at least 800 IOPS, but if you have to index 1TB/day: storage is usually the bottleneck but to index much data you have to use many CPUs and rAM.
Splunk requires:

  • as minimun specifications 12 CPUs and 12 GB RAM,
  • as mid range specifications 24 CPUs and 64 GB RAM,
  • as high performance specifications 48 CPUs and 128 GB RAM.

the question is: how many Indexers you have to use?
Remembering that Indexers work also to answer to searches (users and scheduled).

So to index 600-1000 GB/day you should use:

  • 3 Indexers, if you have up to 8 users,
  • 4 Indexers, if you have up to 16 users,
  • 6 Indexers, if you have up to 24 users,
  • 7 Indexers, if you have up to 48 users,

If you have more than 1TB/day, capacity planning grows, 1-2 TB/day:

  • 7 Indexers, if you have up to 4 users,
  • 8 Indexers, if you have up to 8 users,
  • 10 Indexers, if you have up to 16 users,
  • 12 Indexers, if you have up to 24 users,
  • 14 Indexers, if you have up to 48 users,

Then is also relevant how many scheduled searches you have running, or if you have many apps like Enterprise Security or ITSI that contain many scheduled searches requires additional resources.
A complete training about capacity planning is one of the themes of Splunk Architect Training (that I hint!).

Anyway returning to your question: Splunk requires at least 800 IOPS or more (obviously better!), that means 8x15K rpm SAS drives in RAID 1+0 configuration or SSD disks.
Then analyze your requirements and plan the correct number and resources of your Indexers (and Search Heads).

Ciao.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...