Getting Data In

Retention Time Question

hartfoml
Motivator

I have this setting in my index.conf

frozenTimePeriodInSecs = 48211200

As I understand it this should keep data in the index for about 1.53 years

I do not have any data in the index past January 9th of this year. Somehow the data has rolled out or has it?

Also if the data really has rolled out, then I need to restore from an old copy of the index back from January.

Can you point me to the process doc to connect an old copy of the index and extract the logs (Many Millions of lines - Firewall Logs) and put the logs back into the online index.

Thanks for the help.

Support said they wouldn't be able to help until tomorrow and needed a diag.

A diag? for what, to find out what happened 11 months ago?

How would I check to see what is rolling out of the firewall index?

0 Karma
1 Solution

alacercogitatus
SplunkTrust
SplunkTrust

The file should be "indexes.conf". Where is this file located?

You can check the settings by using: $SPLUNK_HOME/bin/splunk cmd btool --debug indexes list YOUR_INDEX_NAME. This will show you from what file a setting was included, and what all the options are for that index. Do this on the indexers.

If you have really rolled data - you will need to restore the buckets - copy and paste will do as long as the bucket ids don't collide.

EDIT AND UPDATE:

So according to your last comment - you are maxing your index at <1TB, so it will roll off data when the index reaches that threshold. Set your maxTotalDataSizeMB = 104857600. You will then start storing your data up to 1.5 years OR 100 TB, which ever comes FIRST.

View solution in original post

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

The file should be "indexes.conf". Where is this file located?

You can check the settings by using: $SPLUNK_HOME/bin/splunk cmd btool --debug indexes list YOUR_INDEX_NAME. This will show you from what file a setting was included, and what all the options are for that index. Do this on the indexers.

If you have really rolled data - you will need to restore the buckets - copy and paste will do as long as the bucket ids don't collide.

EDIT AND UPDATE:

So according to your last comment - you are maxing your index at <1TB, so it will roll off data when the index reaches that threshold. Set your maxTotalDataSizeMB = 104857600. You will then start storing your data up to 1.5 years OR 100 TB, which ever comes FIRST.

0 Karma

hartfoml
Motivator

Kyle,

You mentoend above that after resoring the buckets I could copy to the db and as long as the bucket ID's don't collide.

Is that right? or do I need to do a ./splunk rebuild $SPLUNK_HOME/var/lib/splunk/firewall/thaweddb/temp_db_1*

And then copy reindexed temp folder to $SPLUNK_HOME/var/lib/splunk/firewall/db/db_1*

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

See my original answer for the answer. You are limiting based on size, not time. Update the conf for the index and it should start storing all the data.

0 Karma

hartfoml
Motivator

There are no files in the $SPLUNK_DB/firewall/colddb

[splunk@Mysystem splunk]$ du -sh firewall/

978G firewall/

I have three indexers with the same size

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

also just hit me, you have "maxTotalDataSizeMB = 1000000", which is less than 1 TB. Not sure about you, but a years worth of firewall data is more than that. What does du -sh $SPLUNK_DB/firewall return?

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

COLDDB - not DB

0 Karma

hartfoml
Motivator

in the $SPLUNK_DB/firewall/db I have these files

drwx------ 3 splunk splunk 8192 Jan 15 2013 db_1357863050_1357725658_118
drwx------ 3 splunk splunk 4096 Jan 17 2013 db_1358072144_1357863051_119
drwx------ 3 splunk splunk 4096 Jan 18 2013 db_1358267635_1358072147_120
drwx------ 3 splunk splunk 8192 Jan 21 2013 db_1358415963_1358267682_121
drwx------ 3 splunk splunk 4096 Jan 23 2013 db_1358556654_1358415964_122
drwx------ 3 splunk splunk 4096 Jan 24 2013 db_1358773965_1358556655_123
drwx------ 3 splunk splunk 8192 Jan 26 2013 db_1358945916_1358773976_12

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

Try these commands and see what returns (run them from $SPLUNK_DB/firewall/colddb😞

touch --date "2013-01-02" /tmp/start
touch --date "2013-01-03" /tmp/end
find . -type f -newer /tmp/start -not -newer /tmp/end

0 Karma

hartfoml
Motivator

Search GUI

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

This shows that your frozen time is correct. How are you determining that the data is not there? Via Search GUI or manually looking in the db files?

0 Karma

hartfoml
Motivator

system rawChunkSizeBytes = 131072
system rotatePeriodInSecs = 60
system serviceMetaPeriod = 25
system suppressBannerList =
system sync = 0
system syncMeta = true
system thawedPath = $SPLUNK_DB/firewall/thaweddb
system throttleCheckPeriod = 15

0 Karma

hartfoml
Motivator

system maxConcurrentOptimizes = 3
system maxDataSize = auto_high_volume
system maxHotBuckets = 3
system maxHotIdleSecs = 0
system maxHotSpanSecs = 7776000
system maxMemMB = 5
system maxMetaEntries = 1000000
system maxRunningProcessGroups = 20
system maxRunningProcessGroupsLowPriority = 1
system maxTotalDataSizeMB = 1000000
system maxWarmDBCount = 300
system memPoolMB = auto
system minRawFileSyncSecs = disable
system partialServiceMetaPeriod = 0
system quarantineFutureSecs = 2592000
system quarantinePastSecs = 77760000

0 Karma

hartfoml
Motivator

system [firewall]
system assureUTF8 = false
system blockSignSize = 0
system blockSignatureDatabase = _blocksignature
system coldPath = $SPLUNK_DB/firewall/colddb
system coldToFrozenDir =
system coldToFrozenScript =
system compressRawdata = true
system defaultDatabase = main
system enableOnlineBucketRepair = true
system enableRealtimeSearch = true
system frozenTimePeriodInSecs = 48211200
system homePath = $SPLUNK_DB/firewall/db
system indexThreads = auto
system maxBloomBackfillBucketAge = 30d

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

What does the output of that command show? What version are you using?

0 Karma

hartfoml
Motivator

/opt/splunk/etc/system/local/indexes.conf

Sorry typing too fast for my own good.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...