Index appears to be truncated prior to Max Index Size

[BradL]

I have an index "eng_1" that has a max size of 500,000 MB. When I look in SplunkOnSplunk it reports this index to be 25% full, however, the oldest data I can query is about 100 days old. Given this index gets about 5GB/day, that amount of history seems right, but the 25% full seems to imply I should be able to go back much further.

| rest /services/data/indexes | search title="eng_1" | table currentDBSizeMB, splunk_server, maxTotalDataSizeMB, maxWarmDBCount, homePath.maxDataSizeMB, coldPath.maxDataSizeMB

yields:

`
currentDBSizeMB splunk_server maxTotalDataSizeMB maxWarmDBCount homePath.maxDataSizeMB coldPath.maxDataSizeMB

1 index1splunk.au1.domain.net 500000 300 300000 200000
1 index1splunk.br1.domain.net 500000 300 300000 200000
7126 index1splunk.eu1.domain.net 500000 300 300000 200000
134915 index1splunk.us1.domain.net 500000 300 300000 200000
`

So on one hand it seems like the amount of history I see is approximately right given the expected max size and the amount of data I ingest every day, but the metadata and SplunkOnSplunk doesn't show that this index is full.

How can I confirm whether the daily incoming data is causing the old data to get evicted?

[BradL · Accepted solution]

Turns out what's happening is the VOLUME as a whole is reaching it's limit. Therefore the indexes residing on the volume get trimmed to keep the volume size in check, while the index itself is only partially full.

[somesoni2]

The data retention of an index is influenced by two factors
maxTotalDataSizeMB - If total size reaches beyond this limit, data buckets will be rolled over to frozen
frozenTimePeriodInSecs -= if latest time for a bucket is older than this period, that data bucket will be rolled over to frozen.

Run the following query and check the value for frozenTimePeriodInSecs if it has been reduced (default is 6 years) to lower values like 100 days in your case.

| rest /services/data/indexes | search title="eng_1" | table frozenTimePeriodInSecs 

[BradL]

Thanks for the reply. I ran the query:

frozenTimePeriodInSecs = 31,536,000 // 1 year = ( 365 x 24 x 3600 )

This is actually what I was expecting.