Getting Data In

Estimating size of blockSignature index

dwaddle
SplunkTrust
SplunkTrust

Is there any data available on estimating how big an indexer's blockSignature database may become, based the blockSignSize and daily event volumes?

Tags (2)

jrodman
Splunk Employee
Splunk Employee

blockSIgnature events are just listings of eventIDs. Therefore a block signature event is just a bunch of numbers like 45:35 with spaces in between. Therefore the size of the block signature database has no correspondence with the size of the original events, making this difficult to estimate in bytes.

However, each event requires a price of around 8 bytes in the text, precompressed, and each event requires a hash and indexing-related overhead. I would guesstimate that each block, in in the current rawdata format and tsidx format, is going to cost on the order of a kilobyte. Empirical data is obviously vastly superior to my guesses.

There's therefore the obvious tunable here which is the blockSignSize. I think anyone using this feature significantly probably wants to tune that value upwards. You don't want it too high for low incoming datarates, but at higher rates, plausibly values like 10000 might be reasonable.

0 Karma

Michael_Wilde
Splunk Employee
Splunk Employee

Good question. Mine is the same size of my main index. Not thrilled.

Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...