Deployment Architecture

Splunk 6.5.1 : Search Head Cluster deployment changing default app.conf in user-prefs violating system-provided install manifest

ranton5000
New Member

I have been trying to clear an alert on a search head cluster that complains about :

File Integrity checks found 1 files that did not match the system-provided manifest. See splunkd.log for details.

Turns out the file is $SPLUNK_HOME/etc/apps/user-prefs/default/app.conf :

01-18-2017 14:42:00.136 +0800 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/apps/user-prefs/default/app.conf" did not pass hash-checking due to reason="content mismatch"

So I went and checked and set it to the standard 6.5.1 default file within the $SPLUNK_HOME/etc/shcluster/apps/user-prefs/default/app.conf on the search head deployment server. ( recently upgraded from 6.3.4 )

Once I run a SH cluster deploy splunk adds the following line to the $SPLUNK_HOME/etc/apps/user-prefs/default/app.conf that gets copied to each search head :

install_source_checksum = a9cff524a35e46b2e2a58a0a0129b3354066e789

Which is different to the mainifest in /opt/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest :
f 444 splunk splunk splunk/etc/apps/user-prefs/default/app.conf ac9ff5d098283488c186e9f7b7464f0e269c332eef70db6f560b9392d6289878

Therefore it's appears to be checksum fault due to file being different from the install file.

Great 😞

Even if you remove the offending line from app.conf the error disappears however the SH deployer overwrites it and error returns.

Does anyone have a workaround and can someone confirm it as a bug ?

Tags (1)
0 Karma

rfaircloth_splu
Splunk Employee
Splunk Employee

user_prefs should not be deployed via deployer remove shcluster/apps/user-prefs, deploy to peers, return to each peer and reinstall the rpm/tar to restore the missing files

A few other things to check
1. review all contents of shcluster/apps ensure install_source_checksum is not present in default|local/apps.conf for any deployed apps if you have to clean up deploy to the cluster after cleanup actions.
2. Make sure the SHC members are not the client of a deployment server, if they are (deploymentclient.conf) remove this file and run a rolling restart. find and remove the deployment client artifacts left in opt/splunk/var

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...