Old events are not searchable

[roopeshetty]

Hi

We are very new to Splunk. We had added some server events for CPU usage values using settings>Data Inputs>Remote performance monitoring almost 50 days back by creating dedicated index of 900GB for them. All was good till yesterday so that we could get all the events from the very first day since we indexed them. But to our surprise, when we checked it yesterday, we could just get the data from last 15 days, but the previous 40 days of events are not showing in searches. Not sure where the old data has gone. And even we verified it in Index tab and it's showing earliest event as just 20 days ago, not 50 days ago. Can some one please help on this by getting these events back and by explaining where these old events have gone?

thanks
roopesh

[ryanoconnor]

I would highly recommend Fire Brigade for this type of scenario. It's main purpose is for troubleshooting issues like this and is a really great app.

https://splunkbase.splunk.com/app/1581/

[somesoni2]

It could be due to the retention policy on the index (the bucket containing the event that you're search might have rolled-off to frozen. Could you post value of attribute 'frozenTimePeriodInSecs' for the index (can find in the indexes.conf on etc/system/local OR under local folder of the app).
Other methods to find this setting.
REST Query (run following search in Splunk)

| rest /services/data/indexes | search title="PutYourIndexNameHere" | table title frozenTimePeriodInSecs| dedup title frozenTimePeriodInSecs

Splunk CLI
bin/splunk cmd btool indexes list putyourIndexName

[roopeshetty]

Hi

we ran the query and got the result as frozenTimePeriodInSecs=188697600. So what does it indicates? can you please explain. And we dont have indexes.conf in etc/system/local instead we have it in etc\system\default path. Please confirm how solve this issue?

regards

[lycollicott]

Can you post the contents of D:\splunk\etc\system\local\index.conf?

[roopeshetty]

Hi
There is no file by name index.conf in D:\splunk\etc\system\local. What files we have on this path are as below;

server.conf
migration.conf
inputs.conf
authorise.conf
alert_action.conf
readme

[roopeshetty]

Hi
Below entries i copied from our slpunk installation path that is D:\splunk\etc\system\default\index.conf

---

Version 6.3.3

DO NOT EDIT THIS FILE!

Changes to default files will be lost on update and are difficult to

manage and support.

Please make any changes to system defaults by overriding them in

apps or $SPLUNK_HOME/etc/system/local

(See "Configuration file precedence" in the web documentation).

To override a specific setting, copy the name of the stanza and

setting to the file where you wish to override it.

This file configures Splunk's indexes and their properties.

"global" params (not specific to individual indexes)

sync = 0
indexThreads = auto
memPoolMB = auto
defaultDatabase = main
enableRealtimeSearch = true
suppressBannerList =
maxRunningProcessGroups = 8
maxRunningProcessGroupsLowPriority = 1
bucketRebuildMemoryHint = auto
serviceOnlyAsNeeded = true
serviceSubtaskTimingPeriod = 30
maxBucketSizeCacheEntries = 0
processTrackerServiceInterval = 1
hotBucketTimeRefreshInterval = 10

index specific defaults

maxDataSize = auto
maxWarmDBCount = 300
frozenTimePeriodInSecs = 188697600
rotatePeriodInSecs = 60
coldToFrozenScript =
coldToFrozenDir =
compressRawdata = true
maxTotalDataSizeMB = 500000
maxMemMB = 5
maxConcurrentOptimizes = 6
maxHotSpanSecs = 7776000
maxHotIdleSecs = 0
maxHotBuckets = 3
quarantinePastSecs = 77760000
quarantineFutureSecs = 2592000
rawChunkSizeBytes = 131072
minRawFileSyncSecs = disable
assureUTF8 = false
serviceMetaPeriod = 25
partialServiceMetaPeriod = 0
throttleCheckPeriod = 15
syncMeta = true
maxMetaEntries = 1000000
maxBloomBackfillBucketAge = 30d
enableOnlineBucketRepair = true
enableDataIntegrityControl = false
maxTimeUnreplicatedWithAcks = 60
maxTimeUnreplicatedNoAcks = 300
minStreamGroupQueueSize = 2000
warmToColdScript=
tstatsHomePath = volume:_splunk_summaries\$_index_name\datamodel_summary
homePath.maxDataSizeMB = 0
coldPath.maxDataSizeMB = 0
streamingTargetTsidxSyncPeriodMsec = 5000
journalCompression = gzip

By default none of the indexes are replicated.

repFactor = 0

[volume:_splunk_summaries]
path = $SPLUNK_DB

index definitions

[main]
homePath = $SPLUNK_DB\defaultdb\db
coldPath = $SPLUNK_DB\defaultdb\colddb
thawedPath = $SPLUNK_DB\defaultdb\thaweddb
tstatsHomePath = volume:_splunk_summaries\defaultdb\datamodel_summary
maxMemMB = 20
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxHotBuckets = 10
maxDataSize = auto_high_volume

[history]
homePath = $SPLUNK_DB\historydb\db
coldPath = $SPLUNK_DB\historydb\colddb
thawedPath = $SPLUNK_DB\historydb\thaweddb
tstatsHomePath = volume:_splunk_summaries\historydb\datamodel_summary
maxDataSize = 10
frozenTimePeriodInSecs = 604800

[summary]
homePath = $SPLUNK_DB\summarydb\db
coldPath = $SPLUNK_DB\summarydb\colddb
thawedPath = $SPLUNK_DB\summarydb\thaweddb
tstatsHomePath = volume:_splunk_summaries\summarydb\datamodel_summary

[_internal]
homePath = $SPLUNK_DB_internaldb\db
coldPath = $SPLUNK_DB_internaldb\colddb
thawedPath = $SPLUNK_DB_internaldb\thaweddb
tstatsHomePath = volume:_splunk_summaries_internaldb\datamodel_summary
maxDataSize = 1000
maxHotSpanSecs = 432000
frozenTimePeriodInSecs = 2592000

[_audit]
homePath = $SPLUNK_DB\audit\db
coldPath = $SPLUNK_DB\audit\colddb
thawedPath = $SPLUNK_DB\audit\thaweddb
tstatsHomePath = volume:_splunk_summaries\audit\datamodel_summary

[_thefishbucket]
homePath = $SPLUNK_DB\fishbucket\db
coldPath = $SPLUNK_DB\fishbucket\colddb
thawedPath = $SPLUNK_DB\fishbucket\thaweddb
tstatsHomePath = volume:_splunk_summaries\fishbucket\datamodel_summary
maxDataSize = 500
frozenTimePeriodInSecs = 2419200

this index has been removed in the 4.1 series, but this stanza must be

preserved to avoid displaying errors for users that have tweaked the index's

size/etc parameters in local/indexes.conf.

[splunklogger]
homePath = $SPLUNK_DB\splunklogger\db
coldPath = $SPLUNK_DB\splunklogger\colddb
thawedPath = $SPLUNK_DB\splunklogger\thaweddb
disabled = true

[_introspection]
homePath = $SPLUNK_DB_introspection\db
coldPath = $SPLUNK_DB_introspection\colddb
thawedPath = $SPLUNK_DB_introspection\thaweddb
maxDataSize = 1024
frozenTimePeriodInSecs = 1209600

[jmallorquin]

Hi,

Can you post the configuration of the index?

[roopeshetty]

Hi

Index configuration is :
Max Size of Entire Index = 900GB,
Max Size of Hot/Warm/Cold Bucket = auto,
home path=$SPLUNK_DB\usslcprodwcf_memory\db
cold path=$SPLUNK_DB\usslcprodwcf_memory\colddb
thawed path= $SPLUNK_DB\usslcprodwcf_memory\thaweddb

[jmallorquin]

Hi,

I mean the configuration in indexes.conf.
Are you sure that you are indexing the data in the index that you have configured?