All Apps and Add-ons

SbrkSysAllocator failed

ttrolf
Explorer

Hi,

since we upgraded to Splunk 4.3 from 4.2 we keep seeing this WARN coming up:

01-13-2012 14:49:35.988 +0100 WARN  ProcessDispatchedSearch - PROCESS_SEARCH src/system-alloc.cc:423] SbrkSysAllocator failed.

We see this popping also in Splunkd.log before the former message kicks in:

01-13-2012 14:44:05.902 +0100 WARN  DistributedBundleReplicationManager - bundle replication to 2 peer(s) took too long (13054ms), bundle file size=28390KB, replication_id=1326462232

We run Splunk 64bit on RHEL5 64bit Linux.

UPDATE: The server seems to allocate large amounts of memory, up to 10Gb and lots of swap space too. Is this some memory bug?

Tags (2)
1 Solution

dwaddle
SplunkTrust
SplunkTrust

It's possible you are seeing this issue: http://splunk-base.splunk.com/answers/37809/halp-consulting-the-summary-dashboard-of-the-search-app-... . Can you please follow its advice and report back? Thanks.

View solution in original post

hodsonc
Explorer

I just got this error while running fsck.

I upgraded to 4.3 and after doing the indexer it told me I should run an fsck on the indexes. After shutting down splunk and starting the fsck I got the above error.

  $ /opt/splunk/bin/splunk fsck --mode metadata --all --repair
bucket=/opt/splunk/var/lib/splunk/_internaldb/db/db_1327351718_1327338541_3791 file='/opt/splunk/var/lib/splunk/_internaldb/db/db_1327351718_1327338541_3791/Sources.data' code=16 contains recover-padding, repairing...
src/system-alloc.cc:423] SbrkSysAllocator failed.

other buckets then continue to be worked on.

0 Karma

hexx
Splunk Employee
Splunk Employee

This might be a different problem, I would suggest that you ask a new question. Some of the stuff we'll want to find out :

  • Does this happen consistently and always with the same bucket?
  • How much physical and virtual memory is available on your system at the time that you get the error?
0 Karma

dwaddle
SplunkTrust
SplunkTrust

It's possible you are seeing this issue: http://splunk-base.splunk.com/answers/37809/halp-consulting-the-summary-dashboard-of-the-search-app-... . Can you please follow its advice and report back? Thanks.

ttrolf
Explorer

Hi following the link you sent me duckfez solved the issue, thank you! Awaiting Splunk 4.3.1 then.

hexx
Splunk Employee
Splunk Employee

I agree with @duckfez I believe the message you are reporting is merely a symptom of a memory exhaustion problem. If you'd like to see if Splunk is responsible for that, I would suggest to install the Splunk on Splunk app and to turn on the ps_sos.sh scripted input to track memory usage by Splunk process (splunkd, splunkweb, searches).

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...