Why is splunkd crashing with error "STMgr out of memory failure" at the same time every day since upgrading to Splunk 6.3.1?

[philippz]

Since i upgraded our Splunk to 6.3.1 we got the error

"ERROR STMgr - dir='......' out of memory failure". 

The last three days we got this error at the same time and index with the result of crashing of the splunkd.exe.

This happens every day at 05:43 am, and mostly it's the index "defaultdb" that gets an "out of memory failure".

splunkd.log:

11-26-2015 05:43:36.878 +0100 ERROR STMgr - dir='E:\Splunk\defaultdb\db\hot_v1_542' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
11-26-2015 05:43:36.878 +0100 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index

We used Splunk 6.3.1 on Windows Server 2008 RC 2

On the following attachment you can see the occurs of this error over the last 7 days count by the dir.

[lrudolph]

Did you get any solution for this from Support? We have the same problem and the whole cluster is crashing almost every day (currently running version 6.3.4.)

[lguinn2]

Does your Splunk server meet the specifications for running Splunk?

For Windows, the recommended specifications are
2x six-core, 2+ GHz CPU
12GB RAM
RAID 0 or 1+0
64-bit OS installed

This comes from the Installation manual, under System Requirements

If yes, then the next question is "What searches were running when Splunk crashed?" In particular, I would look at scheduled searches, since you say that this is happening at the same time every day.

FYI, the "defaultdb" is the main index.

Finally, if you have Splunk support, this is the sort of problem that should be reported!

[philippz]

Thansk for your anwser.

Yes our Splunk server meet the specifications. This couldn't be the problem.

There is also no scheduled search which could be the reason for the crash.

So i will open a ticket in splunkt support.

thanks

[LewisWheeler]

Did you ever find a solution to this problem?