Knowledge Management

configuration file for index and summary index

shaganga
New Member

Hi

we need your help in creating the configuration to align the requirements.
we have created index for application logs rpp_pe_idx_dmc and we have created schedule saved search to perform some searches and store the results by enabling summary index at rpp_pe_summary_idx_dmc. Question here is we need to update the indexes.conf to meet below requirements.

  • Hot&Warm buckets will have 90 days of raw data (for index rpp_pe_idx_dmc)
  • Cold buckets will have last 10 months of summary data (for index rpp_pe_summary_idx_dmc)

If we look at my incomplete indexes.conf:

[rpp_pe_idx_dmc]
coldPath = volume:COLD/rpp_pe_idx_dmc/colddb
homePath = volume:HOTWARM/rpp_pe_idx_dmc/db
thawedPath = $SPLUNK_DB/rpp_pe_idx_dmc/thaweddb

[rpp_pe_summary_idx_dmc]
coldPath = volume:COLD/rpp_pe_summary_idx_dmc/colddb
homePath = volume:HOTWARM/rpp_pe_summary_idx_dmc/db
thawedPath = $SPLUNK_DB/rpp_pe_summary_idx_dmc/thaweddb

could you provide us the completed configuration of those two snippets to meet the requirements.

Thanks !!

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi shaganga,
let me understand:
you said that retention of rpp_pe_idx_dmc is 90 days in hot/warm data, but what is the retention of cold data?
do you want to use summary only for cold data, not also for hot/warm data? why?
It's not clear your requirement: how long do you want to archive full logs? in other words what is the retention?
How do you want to use summary: for archive summary data or to accelerate searches?

Anyway you can define:

  • the max number oh warm buckets: maxWarmDBCount = ,
  • The maximum size of an index (in MB): maxTotalDataSizeMB = ,
  • Total retention period: frozenTimePeriodInSecs = ,
  • The maximum size in MB for a hot DB to reach before a roll to warm is triggered: maxDataSize = |auto|auto_high_volume,
  • Maximum hot buckets that can exist per index: maxHotBuckets = ,
  • The maximum size of homePath (which contains hot and warm buckets): homePath.maxDataSizeMB = ,
  • The maximum size of coldPath (which contains cold buckets): coldPath.maxDataSizeMB = For full information see http://docs.splunk.com/Documentation/Splunk/7.0.0/Admin/Indexesconf.

Bye.
Giuseppe

0 Karma

shaganga
New Member

Hi @Giuseppe

Thanks for quick response. If at all we require to keep raw index for 3 months retention and summary index for 13 months retention. Could you please advise the how configuration looks like?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi shaganga,
to set the retention period of an index you have to put in the related stanzas of your indexes.conf the following row

frozenTimePeriodInSecs = <integer>

So if you have a row index called my_index with a retention of 90 days and a summary index called my_summary with a retention of 13 months (395 days) you have to insert:

[my_index]
frozenTimePeriodInSecs = 7776000

[my_summary]
frozenTimePeriodInSecs = 34128000

obviously remember that a bucket will be deleted when the latest event of the bucket will be out of retention period, so the earliest events of a bucket will remain online more than the retention period.

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...