Getting Data In

Estimating size of blockSignature index

dwaddle
SplunkTrust
SplunkTrust

Is there any data available on estimating how big an indexer's blockSignature database may become, based the blockSignSize and daily event volumes?

Tags (2)

jrodman
Splunk Employee
Splunk Employee

blockSIgnature events are just listings of eventIDs. Therefore a block signature event is just a bunch of numbers like 45:35 with spaces in between. Therefore the size of the block signature database has no correspondence with the size of the original events, making this difficult to estimate in bytes.

However, each event requires a price of around 8 bytes in the text, precompressed, and each event requires a hash and indexing-related overhead. I would guesstimate that each block, in in the current rawdata format and tsidx format, is going to cost on the order of a kilobyte. Empirical data is obviously vastly superior to my guesses.

There's therefore the obvious tunable here which is the blockSignSize. I think anyone using this feature significantly probably wants to tune that value upwards. You don't want it too high for low incoming datarates, but at higher rates, plausibly values like 10000 might be reasonable.

0 Karma

Michael_Wilde
Splunk Employee
Splunk Employee

Good question. Mine is the same size of my main index. Not thrilled.

Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...