Splunk app Windows Infrastructure - kvstore disk saturations

[mabonjean]

Hi,

I use on my Search Head Cluster (with 80GB of disk space for each SH) the application "Splunk App Windows Infrastructure" that's carrying several kvstores and collections configurations.

Theses KVStore consummed all the disk space. I must disable several scheduled tasks. Mainly all kvstore and lookups updates tasks.

My problem is still here. This app still consummed my disk space and the KVStore doesn't rotate old data.
I won't clean it to prevent losts of valuables datas.

How can I modify / optimize the configuration to stop the high disk consummation ?

Best regards.

[nick405060]

The answer is that you likely have both the Exchange and Windows app installed, which is the issue. To me this is something that should be addressed ASAP, it seems like a very serious issue that any Splunk customer that installs both apps gets their disk space blown up (unsure which versions of Splunk this affects):

https://docs.splunk.com/Documentation/MSExchange/4.0.0/DeployMSX/Platformandhardwarerequirements#Do_...

I ran this per @dwaddle:

|  rest splunk_server=local /services/server/introspection/kvstore/collectionstats 
|  fields data 
|  mvexpand data 
|  rename data as _raw 
|  spath 
|  fields - _raw
|  fields ns size storageSize totalIndexSize

And it showed me that the terminal service trackers are the culprits:

splunk_app_microsoft_exchange.tSessions_collection and splunk_app_windows_infrastructure.tSessions_collection

Per @automine, app savedsearches likely have the same name and are using the same collection. You can disable tSessions_Lookup_Update* savedsearches in the apps, or, like the Documentation link says, just uninstall the Windows Infra app.

This is also the same issue:

https://answers.splunk.com/answers/716097/kvstore-mongo-consuming-40gb-space.html

Oh also a local clean of the kvstore should clear it out

[mabonjean]

Hi Nick,

Thanks for your reply.

We solve (I expect permanently) my issue with cleaning all local KVStore.

The cause is somes old lookups files (old from months ago) can't be delete by Splunk.
When I clean a local KVstore, the non replicated old lokkup files are replicate after the cleaning.

I'll check you answer and make sur that all is ok with my apps.
I'll make a return after.

Thank you.