Deployment Architecture

WARN StreamedSearch - Could not find bundles for search head provided checksum=

rphillips_splk
Splunk Employee
Splunk Employee

The message: WARN StreamedSearch - Could not find bundles for search head provided checksum=xxx keeps popping up on one search head in a search head pool from all search peers anytime a search (basic or complex) from that search head is initiated. Verified that all search heads, nfs server and indexers are synchronized to the same ntp server.

SPLUNK VERSION:
VERSION=6.1.3
BUILD=220630
PRODUCT=splunk
PLATFORM=Linux-x86_64

1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

I recently encountered this issue and did not see any answers on how to resolve this message other than check ntp sync. but what if ntp is already in sync between search heads, nfs and indexers?

-check status of your search peers from the search head throwing the WARN message and see if any are in a failed state.
settings> distributed search> Search peers

(in this case one of the indexers had replication status of failed)

Although not a sophisticated answer this is what was done to correct / stop the message from occurring:

  • restart splunk on the indexer that had replication status of failed
  • restart splunk on the search head throwing the WARN message
  • verify the indexer replication status is successful via the Search Head GUI >settings> distributed search> Search peers

to restart splunk from command line:
$SPLUNK_HOME/bin
./splunk restart

subsequently another set of messages (below) suspected to be related to the problem we were seeing also cleared:

ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/xxx.bundle

WARN DistBundleRestHandler - There was a problem renaming: /opt/splunk/var/run/searchpeers/xxx.tmp -> /opt/splunk/var/run/searchpeers/xxx: Directory not empty

View solution in original post

rsimmons
Splunk Employee
Splunk Employee

This was a known issue (SPL-97601) in bundle replication where skewed modtimes on temporary bundle files cause premature reaping and errors in distributed search. The workaround is to fix clock skew between indexers and NFS server hosting $SPLUNK_HOME/var/run/searchpeers. The issue has also been resolve in the latest release of Splunk.

rphillips_splk
Splunk Employee
Splunk Employee

I recently encountered this issue and did not see any answers on how to resolve this message other than check ntp sync. but what if ntp is already in sync between search heads, nfs and indexers?

-check status of your search peers from the search head throwing the WARN message and see if any are in a failed state.
settings> distributed search> Search peers

(in this case one of the indexers had replication status of failed)

Although not a sophisticated answer this is what was done to correct / stop the message from occurring:

  • restart splunk on the indexer that had replication status of failed
  • restart splunk on the search head throwing the WARN message
  • verify the indexer replication status is successful via the Search Head GUI >settings> distributed search> Search peers

to restart splunk from command line:
$SPLUNK_HOME/bin
./splunk restart

subsequently another set of messages (below) suspected to be related to the problem we were seeing also cleared:

ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/xxx.bundle

WARN DistBundleRestHandler - There was a problem renaming: /opt/splunk/var/run/searchpeers/xxx.tmp -> /opt/splunk/var/run/searchpeers/xxx: Directory not empty

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...