Splunk 6.3 Capacity gains by 20%. Any technical information on what changed?

[Lucas_K]

In the 6.3 keynote, it was mentioned that Splunk 6.3 "increased capacity by 20%".

"And increase the overall capacity of your deployment by 20% or more" - http://blogs.splunk.com/2015/09/22/splunk-enterprise-6-3-shaking-it-up/

It is mentioned in several different places so doesn't seem to be a mistake.

There is no technical information on how this is obtained. It is by improvements in the tsidx structure/rawdata compression? Is it an option depending on your architecture? conf setting or just a new default for new data? standard dist search indexers vs clustered indexers? Replication options chosen?

I need to know so I can best present forward looking splunk architectural designs which were based on older observed compression ratios of the customer's existing data.

[tanwar_splunk · Accepted solution]

The daily indexing guideline per indexer increased from 250 GB/day to 300 GB/day, i.e a 20% increase in indexing volume. This is a result of the optimizations made in 6.3. In order to get a good understanding of some of the improvements and the effect it will have on resource utilization, I recommend going through the session notes from "Harnessing Performance and Scalability in Splunk 6.3" and Splunk docs. Please find the links below

https://conf.splunk.com/session/2015/conf2015_ANekkanti_SPal_ATameem_Splunk_SplunkClassics_Harnessin...

http://docs.splunk.com/Documentation/Splunk/6.3.0/Capacity/Parallelization

http://docs.splunk.com/Documentation/Splunk/6.3.0/Indexer/Pipelinesets

[Lucas_K]

yeah i figured out that it was daily indexing capacity as opposed to storage capacity.

I was hoping that compression methods used had improved to reduce the clustering replication requirements.

[woodcock]

I was at .conf where they demonstrated this and although they did not give any complete explanation, the gist was clear that it did not have to do with the software data architecture/design but rather how CPU is allocated to the various distinct processes that the Indexers run. It seemed like the "dedicated-CPU-core" situation is no longer the case and that everything is threaded in such a way that CPUs don't get fully locked to a particular process (e.g. Real-Time). I would take a look at "limits.conf" and assume that there are new settings there related to the changes they made.

[echalex]

Hi Lucas K,

AFAIK, the gains have been mostly achieved through improved usage of multiple cores. If you check the release notes, you'll see that it mentiones optimized CPU utilization both for indexing and searching.

http://docs.splunk.com/Documentation/Splunk/6.3.0/ReleaseNotes/MeetSplunk

[Lucas_K]

ok I think i might have read this wrong.

It might not be referring to storage capacity rather indexing capacity ... ie. 333gb/day per index vs 250gb/day per index on the same hardware.