Splunk Search

Will having lots of extracted fields increase my index size?

Mick
Splunk Employee
Splunk Employee

I need to understand how adding fields to raw data will increase our index size growth. We are in the process of adding many fields to individual records. This could run to as much as hundreds per record / line.

Will each record contain a field label and a repeat of the data from the full raw record? This could cause the increase in our index size to be very substantial in extracting fields.

Tags (2)
1 Solution

Mick
Splunk Employee
Splunk Employee

Generally, the vast majority of field extractions are performed at search time, so configuring lots of fields per event will not increase your index size by any amount. It is possible to create index-time extractions, so that extra fields are written to the index but that has to be specifically configured and is only recommended in isolated cases where it's absolutely necessary for search-speed to be as fast as possible.

The IFX feature, rex command and regular 'EXTRACT' settings in props.conf all work at search-time. Unless you have purposely configured all of your fields to be created at index-time, they will not increase your index size.

View solution in original post

Mick
Splunk Employee
Splunk Employee

Generally, the vast majority of field extractions are performed at search time, so configuring lots of fields per event will not increase your index size by any amount. It is possible to create index-time extractions, so that extra fields are written to the index but that has to be specifically configured and is only recommended in isolated cases where it's absolutely necessary for search-speed to be as fast as possible.

The IFX feature, rex command and regular 'EXTRACT' settings in props.conf all work at search-time. Unless you have purposely configured all of your fields to be created at index-time, they will not increase your index size.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...