Deployment Architecture

Searchhead Pooling: Failed to lock sentinel.txt while saving/deleting search from GUI

bwooden
Splunk Employee
Splunk Employee

Red banner message in GUI (below) regarding sentinel.txt file lock is preventing updates from the GUI.

Error fixing dangling data: Failed to
lock
/mnt/search_head_pool/etc/apps/sentinel.txt
with return code 1: Success

I do not presently have access to the logs of the system (Splunk 4.2.2 101277 on RHEL).

  • Can sentinel.txt and sentinel.txt.lock safely be deleted?
  • Are there known causes/resolutions for this error?
  • What is the scope/impact of this state?
1 Solution

ewoo
Splunk Employee
Splunk Employee

Yes, you can delete these files while Splunk is stopped. They are re-created on demand.

If you see a "stale" sentinel.txt.lock file remaining while Splunk is stopped, that is probably the source of this error.

What is the output of "splunk pooling validate"?

View solution in original post

ewoo
Splunk Employee
Splunk Employee

Yes, you can delete these files while Splunk is stopped. They are re-created on demand.

If you see a "stale" sentinel.txt.lock file remaining while Splunk is stopped, that is probably the source of this error.

What is the output of "splunk pooling validate"?

ewoo
Splunk Employee
Splunk Employee

The most common reasons are mentioned in the comment immediately preceding yours: 1) stale lock file (caused by a crash, for example), or 2) poor performance of shared storage, leading to slow I/O and contention on the lock file.

Some improvements to splunkd were made to reduce the amount of I/O we perform against sentinel.txt; these improvements landed in 5.0.6 and 6.0 (SPL-66563)

0 Karma

splunkIT
Splunk Employee
Splunk Employee

@ewoo, in what circumstances this "failed to lock sentinel.txt" error will occur?

0 Karma

ewoo
Splunk Employee
Splunk Employee

The error is displayed if:

1) the user triggers an action that requires writing to a conf file, and
2) the write fails when the user cannot acquire a file-based mutex

You can't suppress the error. The underlying failure must be addressed -- remove stale lock files, investigate contention on the lock file and/or performance of shared storage, etc.

0 Karma

the_wolverine
Champion

Why would this error be displayed to the user? Can it be suppressed?

0 Karma

ewoo
Splunk Employee
Splunk Employee

This file is only created/used when pooling is enabled.

The file itself acts as the synchronization mechanism for conf writes. In other words, a member of the SHP must "own" this lockfile in order to make conf changes. If a member of the pool X finds the lockfile already owned by another member Y, X will wait for Y to relinquish ownership of the lockfile.

rmorlen
Splunk Employee
Splunk Employee

How does Splunk handle this file in Pooling mode? Which server gets the "lock"? What happens when multiple servers need to lock the file?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...