The above values didn't work for me for some reason, When I used for example: SYSTEM,(?:[^,]*,){18}([^,]*) ....it worked - (same REGEX as yours, but with the suffix after the "}" being different) ... View more

I have to admit that I may have bitten off more than I can chew here - This became way more complex than I thought very quickly. I really appreciate the responses provided here! I am trying to capture what I've gleaned from this in an indexes.conf file. I would appreciate if you can see the comments and questions and perform a sanity check to let me know if I'm missing the plot somewhere   [default] # should these paths be defined in each index stanza, or is it ok in default only? homePath = volume:faststorage/$_index_name/db coldPath = volume:slowstorage/$_index_name/colddb thawedPath = volume:faststorage/$_index_name/thaweddb # Any path required for the Splunk Summaries volume? maxWarmDBCount = 300 # Is this required? If so, is it in the correct place? maxDataSize = auto_high_volume # Is this required? If so, is it in the correct place? frozenTimePeriodInSecs = 64800000 # 2 years [volume:faststorage] path = D:/Splunk/var/lib/splunk # Forward slashes on windows? maxVolumeDataSizeMB = 5242880 # 5TB (no less than amount of 100GB indexes multiplied by 50 non-internal indexes) [volume:slowstorage] path = //MYNAS/Splunk_data maxVolumeDataSizeMB = 10485760 # 10TB # Data is deleted when the volume reaches maximum size, or when frozenTimePeriodInSecs is met - whichever happens first? [volume:_splunk_summaries] path = D:/Splunk/var/lib/splunk # Forward slashes on windows? maxVolumeDataSizeMB = 1048576 # 1 TB [myindex1] homePath.maxDataSizeMB = 102400 #100GB per index [myindex2] homePath.maxDataSizeMB = 102400 #100GB per index [myindex3] homePath.maxDataSizeMB = 102400 #100GB per index #... [myindex50] homePath.maxDataSizeMB = 102400 #100GB per index #Can a wildcard stanza be used such as [myindex*] to keep things simpler? ... View more

I see in https://docs.splunk.com/Documentation/Splunk/9.0.4/admin/Indexesconf#GLOBAL_SETTINGS that the homePath.maxDataSizeMB setting is a per index setting, and not a global setting. Does this mean I have to create separate stanzas in indexes.conf for each index and I cannot use this setting in the default stanza? If a can use it in the default stanza, does it still apply per-index? ... View more

@PickleRick wrote: OK. So you have some indexes using volume definition and some which do not use it which makes it inconsistent and it might not work as you'd expect. You should redefine your default indexes to use volume definition as we discussed with @isoutamo (but be wary of the actual directory name since they don't map 1:1 from index name to directory name!). Ha, I think you are overestimating my knowledge of Splunk 🙂 I'm gonna need some guidance on how to do this please Another thing is check your buckets - especially the timerange of events contained there - with dbinfo and manually verify if they should be frozen or not. Remember that - as I wrote before - index is rolled to frozen if _latest_ event in the index is older than threshold. I tried googling where to find dbinfo and can only find references stating it should be a dashboard. Do you mean dbinspect? Anyway, you have 8TB (I suppose, minus filesystem structures, minus some other system stuff) for 50 indexes for which you want to have 100GB threshold. Remember that unless you explicitly defined them somewhere else, you have additional stuff that is stored with the index, like summaries. And I didn't notice that before - you have your volumes defined but do you have volume limits set? I do not have volume limits. Is this done with maxTotalDataSizeMB? Does this mean that some data may be rolled to frozen even before the frozenTimePeriodInSecs age is reached if the index exceeds the volume limit? Are you absolutely sure that you want tsidx reduction? No I'm not 🙂 I've taken that off I think I need a fresh start with indexes.conf file, I've done a few attempts, but I'm cautious of doing something that will make me lose my data. Can you give me suggestions of what to change in which stanzas please? ... View more

This is what I have in indexes.conf at the moment [default] homePath = volume:faststorage/$_index_name/db coldPath = volume:slowstorage/$_index_name/colddb thawedPath = $SPLUNK_DB/$_index_name/thaweddb maxWarmDBCount = 300 frozenTimePeriodInSecs = 64800000 maxDataSize = auto_high_volume homePath.maxDataSizeMB = 102400 #100GB per index enableTsidxReduction = true timePeriodInSecBeforeTsidxReduction = 15780000 # 6 months [volume:faststorage] path = $SPLUNK_DB [volume:slowstorage] path = //MYNAS/Splunk_data I have not been able to see any evidence that it is actually doing anything.    This is what I had before and it worked, but for some reason not automatically - it would only start rolling when splunk service is restarted and the mysteriously stop a while later, then when restarting splunk it would continue. Once a month I'd have to go in and over the course of a few days restart the splunk indexer service several times to get it to free up enough space on the local drive. [default] homePath = volume:faststorage/$_index_name/db coldPath = volume:slowstorage/$_index_name/colddb thawedPath = $SPLUNK_DB/$_index_name/thaweddb homePath.maxDataSizeMB = 100000 #100GB per index coldPath.maxDataSizeMB = 200000 #200GB per index maxWarmDBCount = 10 frozenTimePeriodInSecs = 64800000 maxDataSize = auto_high_volume [volume:faststorage] path = $SPLUNK_DB [volume:slowstorage] path = //MYNAS/Splunk_data   Of course I restarted splunk after making changes to indexes.conf. @PickleRick See attached screenshots of the btool and rest outputs       Can anyone propose improvements to my indexes.conf I can try which follows all aforementioned best practices and perform the bucket rotating /retention?  Thanks again, appreciate all the responses ... View more

Of course I read that 🙂  I have tried numerous times but I cannot seem to get it the way I would like per the previously mentioned parameters I currently have the following under default stanza.   maxWarmDBCount = 300 frozenTimePeriodinSecs = 31536000 maxDataSize = auto_high_volume homePath.maxDataSizeMB = 102400   But the above does not do what I need. I only have one indexer, so no concerns there.  Could you give me an example of how I should either fix the above, or the indexes.conf file as a whole? ... View more

I'd also like this functionality without having to edit the bin file... Any update on this to allow users to customize their messages? ... View more