Indexer's $SPLUNK_HOME /var/run/searchpeers/ excessive disk usage and bundles not being reaped

[rphillips_splk]

Problem:
Excessive disk space consumed on indexer in $SPLUNK_HOME/var/run/searchpeers to the point where the indexer runs out of disk space. It appears that the bundle files are not being reaped.

[duartet]

Happened here too with Splunk 6.5.5

[rphillips_splk · Accepted solution]

If you are seeing this problem it is likely that you are hitting bug SPL-140831 "Splunk not cleaning up $SPLUNK_HOME/var/run/searchpeers of .delta files and matching directories whose only non-empty subdirectory has the .index extension". This bug affects 6.5.x+ and will be fixed in 6.5.6+ and 6.6.3+

The issue is that the bundle reaper deletes the bundle directory but lookup file indexing recreates it and the .delta bundle files don't get reaped after that. The bundle files will start accumulating on the indexer in $SPLUNK_HOME/var/run/searchpeers/ and can eventually fill up the disk.

To work around the issue increase max_memtable_bytes on the indexers limits.conf to be greater than the largest lookup .csv file in a given .bundle file in /var/run/searchpeers.

on indexers:
$SPLUNK_HOME/etc/system/local/limits.conf

max_memtable_bytes =
* Maximum size, in bytes, of static lookup file to use an in-memory index for.
* Lookup files with size above max_memtable_bytes will be indexed on disk
* A large value results in loading large lookup files in memory leading to bigger
process memory footprint.
* Caution must be exercised when setting this parameter to arbitrarily high values!
* Default: 10000000 (10MB)

This change requires a restart of the indexers

If you have lookup .csv files larger than 10MB they will be indexed to disk on the indexer. Increasing from the default 10MB to a value larger than your largest lookup file will cause the file to be loaded into memory for the period a search process is using that lookup. If you do not need to send the lookup file in the bundle you can blacklist it via distsearch.conf on the search heads https://docs.splunk.com/Documentation/Splunk/6.6.2/DistSearch/Limittheknowledgebundlesize
Lookup files are required to be sent to the indexer if you are using automatic lookups (https://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Knowledge/Makeyourlookupautomatic)
or if you have data model accelerations enabled which are referencing the lookup in the DMA search.

How do I know if my DMA search is referencing a lookup ?

run the following search and you will see the search the data model acceleration is using and if the search contains a lookup reference:

| rest splunk_server=local /services/datamodel/acceleration| fields title search | eval contains_lookup=if(like(search, "%lookup%"),1,0) | eval contains_lookup=case(contains_lookup=1,"yes",contains_lookup=0,"no")| table title search contains_lookup | search contains_lookup=yes

Note: this search does not cover automatic lookups or lookups that might be referenced via a macro

If you see a lookup reference in the search, you need to send that lookup to the indexers in the knowledge bundle so don't blacklist it.