OS Linux
Splunk Version 6.3.1
After executing the following command on my deployer instance, I am seeing this error.
-bash-3.2$ ./splunk/bin/splunk apply shcluster-bundle -target https://myhost@domain.com:8089
Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members. Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
Error while creating deployable apps: Error copying src="/opt/splunk/etc/shcluster" to tmp_staging_area="/opt/splunk/var/run/splunk/deploy.8e9ed66fdbd0ac3d.tmp": No such file or directory
The temp_staging_area does exist and permissions are okay.
-bash-3.2$ ls -ltr
total 376
drwx------ 2 splunkd splunkd 4096 Oct 30 04:45 upload
drwx--x--- 4 splunkd splunkd 4096 Oct 30 04:45 appserver
drwx------ 4 splunkd splunkd 4096 Oct 30 04:46 scheduler
drwx------ 2 splunkd splunkd 4096 Nov 5 21:48 typeahead
drwxr-x--- 4 splunkd splunkd 4096 Dec 1 00:39 deploy.fe8c0cae1fa6cd18.tmp
drwx------ 2 splunkd splunkd 4096 Dec 1 00:47 merged
-rw------- 1 splunkd splunkd 6 Dec 1 00:47 conf-mutator.pid
-rw------- 1 splunkd splunkd 10937 Dec 1 00:47 composite.xml
drwxr-x--- 4 splunkd splunkd 4096 Dec 1 00:48 deploy.c2659c1df562d295.tmp
drwxr-x--- 4 splunkd splunkd 4096 Dec 1 00:49 deploy.39dbf1c7b7c94f85.tmp
drwx------ 4 splunkd splunkd 4096 Dec 2 20:50 deploy
drwxr-x--- 4 splunkd splunkd 4096 Dec 2 20:53 deploy.41b88613e6ab63d0.tmp
drwxr-x--- 4 splunkd splunkd 4096 Dec 2 20:56 deploy.8e9ed66fdbd0ac3d.tmp
drwx--x--x 1309 splunkd splunkd 311296 Dec 2 20:58 dispatch
drwx------ 2 splunkd splunkd 4096 Dec 2 20:58 srtemp
-rw-r----- 1 splunkd splunkd 30 Dec 2 21:00 splunkd.pid
I also have verified that the apps and users directory existing in the deploy.8e9ed66fdbd0ac3d.tmp folder.
-bash-3.2$ cd deploy.8e9ed66fdbd0ac3d.tmp
-bash-3.2$ pwd
/opt/splunk/var/run/splunk/deploy.8e9ed66fdbd0ac3d.tmp
-bash-3.2$ ls -ltr
total 88
drwx--x--- 1963 splunkd splunkd 81920 Dec 2 20:56 users
drwxr-x--- 32 splunkd splunkd 4096 Dec 2 20:56 apps
I had apps symlinked in shcluster/apps...
Splunk support with confirmation of dev says the symlinks would cause issues, so I have removed them and now works fine.
Stray emacs temp files can apparently cause this sort of error as well. (as of 8.2.x). E.g.
.../default/data/ui/views/.#example.xml
just had the same problem with our Deployer. Evidently, an app had multiple files exceeding 256 characters. Errors reported in splunkd.log regarding tar: Path/name too long
To view your name + path limits:
$getconf NAME_MAX /opt/splunk
255
$ getconf PATH_MAX /opt/splunk
4096
I wonder if this is the same behavior for deployment server? Would you experience issues if you symlinked the deployment_apps?
Symlinks are no issue here, but you happen to have one "pointing" to a non-existing directory (dereferenced) from shcluster/apps to apps and/or the symlink itself doesn't have the right permissions.
I had apps symlinked in shcluster/apps...
Splunk support with confirmation of dev says the symlinks would cause issues, so I have removed them and now works fine.
Try doing a "chown -R splunkd:splunkd" on the entire Splunk install tree on the deployer. I had a similar error (and the SPL is probably my complaint) about vague error messages that was a caused by a single bad file in a completely unassociated directory.
I had the same problem and this fixed it for me (no symlinks involved);
Doing "chown -R" on the ../etc/shcluster/apps folder did the trick.
Thx!
it worked for me the same way chown -R the shcluster directory.
Knowing this is an old issue, I had a similar problem on Splunk 7.0.x with the users directory going missing within the shcluster path.
It was not successful. To add a little more context about the shcluster/apps directory, I have symlinked custom apps that relate to a local git repo that I have keeping the apps up to date. Splunk dev confirmed that symlinks would cause this issue.
Ok, at least we have confirmation of an issue and a bugID.
Support Case 293707
I was told SPL-106728 was created to have more logging for the deployer instance.