Knowledge Management

Frequent KVstore crashes due to disk full error from mongod service while disk utilization is at 61%

myerasi
New Member

We have a six search-heads cluster and kv store is frequently crashing on one of them. The error we get is "KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details". And when we look at the mongod.log, we had noticed the following disk full error: 2019-07-10T14:32:21.049Z I CONTROL [journal writer] LogFile::synchronousAppend failed with 8192 bytes unwritten out of 8192 bytes; b=0x55ebc4348000 errno:28 No space left on device
2019-07-10T14:32:21.050Z I - [journal writer] Fatal Assertion 13515

The disk utilization on the host is for the root partition / is at 61%. We did try cleaning up kvstore locally and resyncing it. The servers holds it for around 12 hours after every instance of cleanup and then kvstore crashes again.

Any help in this would be appreciable.

Thanks.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...