splunk is down after upgrade

[smolcj]

hi,
i had 2 splunk instance in one VM, when i tried to upgrade one of the version with ports 9000 and 7089, the other instance which was using the splunk default ports is not turning up.
there isn't any option to start the server, in home folder i could see only
etc infa share var
these folders, bin folder is not seen
please help
did i do anything wrong while upgrading?? i used .rpm fle to upgrade and the command used was
rpm -U --prefix=/opt/splunknewuser splunk-5.0.2-149561-linux-2.6-x86_64.rpm
because already there was an instance of splunk in /opt/splunk
i know that some more info is needed to solve this, but i don't know, what info i have to provide, please ask fpr further explanation specifically..
after i tried copying bin,lib and etc folers from a backup file i got this following error
Splunk> Take the sh out of IT.

Checking prerequisites...
    Checking http port [8000]: open
    Checking mgmt port [8089]: open
    Checking configuration...  Done.
    Checking index directory...  Done.
    Checking databases...
    Validated databases: _audit, _blocksignature, _internal, _thefishbucket, history, infa, main, sample, splunklabs, summary

An unforeseen error occurred:

    Exception: <type 'exceptions.OSError'>, Value: [Errno 13] Permission denied

Traceback (most recent call last):
 File "/opt/splunk/lib/python2.6/site-packages/splunk/clilib/cli.py", line 1082, in main
parseAndRun(argsList)
File "/opt/splunk/lib/python2.6/site-packages/splunk/clilib/cli.py", line 940, in parseAndRun
retVal = cList.getCmd(command, subCmd).call(argList, fromCLI = True)
  File "/opt/splunk/lib/python2.6/site-packages/splunk/clilib/cli.py", line 264, in call
return self.func(args, fromCLI)
    File "/opt/splunk/lib/python2.6/site-packages/splunk/clilib/control_api.py", line 33, in wrapperFunc
return func(dictCopy, fromCLI)
    File "/opt/splunk/lib/python2.6/site-packages/splunk/clilib/_internal.py", line 234, in preFlightChecks
checkSearchthing()
   File "/opt/splunk/lib/python2.6/site-packages/splunk/clilib/_internal.py", line 291, in checkSearchthing
retCode = comm.runAndLog(["locktest"], logStdout = False)
File "/opt/splunk/lib/python2.6/site-packages/splunk/clilib/cli_common.py", line 1151, in runAndLog
proc = subprocess.Popen(cmdList, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "/opt/splunk/lib/python2.6/subprocess.py", line 621, in __init__
errread, errwrite)
File "/opt/splunk/lib/python2.6/subprocess.py", line 1126, in _execute_child
raise child_exception
 OSError: [Errno 13] Permission denied


Please file a case online at http://www.splunk.com/page/submit_issue

[oletigopi]

You might be upgrading it in wrong way.

Ex: For Heavyforwarder 4.3.3 , we can not upgrade directly to 6.2.2
We have to first upgrade to 5.0.3 and then to 6.2.2

[Drainy]

Well I never...
I take it back, it turns out that running an upgrade will delete files from another instance...

I have three instances on one machine. Two instances were running, I ran the upgrade against splunk1 whilst splunk1 and splunk2 were running. The upgrade successfully upgrades splunk1 to the new version, splunk2 however was deleted with only etc, lib, share and var remaining...

Looks like it may be an unintended side-effect of the RPM install process, I suspect its not just working off the prefix and is doing something funky with the running processes. Anyway, in summary, yes. It is all your fault that your colleagues instance has been deleted 🙂

[cramasta]

How would one perform a upgrade in this type of situation without messing up another installation?

[smolcj]

one more issue i felt is like, when i was trying to download .tgz files, it is downloaded as .gz file, but before version 5.0.1 used to download as .tgz

[Drainy]

Oh you've probably completely botched it by copying folders around like that. It depends how those folders were originally created, the user you copied them with, the user that was used to start splunk. Just install from fresh and re-upload your custom apps.

[smolcj]

Thanks drainy, u mean instead of copying those files (bin,include and lib)to splunk, i should reinstall splunk instance and upload the app?
because i got the abovee error after copying those files and trying to run

[Drainy]

Restore from a backup. As far as I can tell they've actually been deleted so short of data recovery, you've just got backups to fall back on.
I have a support case with Splunk and will update if anything more comes back, for now just stick to tar upgrades

[smolcj]

what should i do to restore it? anything that i can do for it?

[Drainy]

Did you run the upgrade as root and then try to run Splunk as the Splunk user?

[Drainy]

In that case I'd say almost certainly that this is nothing to with the upgrade, something else has deleted those files. You'd need to look at other things on the system but this isn't a Splunk issue. Also you are missing an includes folder and I don't recognise the infa directory, something else has happened here.

[smolcj]

exactly.. this instance is in 4.x(the one which is not able to start because i could see only the folders
etc infa share var
bin folder and lib folder was missing and i tried copy those folders from one backup directory i found)
the other one was in 5.0.1 and upgraded to 5.0.2

[Drainy]

Lets just be clear. You have two instances. Installed to completely different folders? e.g. /opt/splunk1/splunk/ and /opt/splunk2/splunk (by default the prefix term on RPM will still install to a splunk directory in the target). So splunk1 and splunk2 work perfectly. you run an upgrade pointed at splunk1. It works, but you suddenly discover that half the files from splunk2 are deleted?

[smolcj]

anything that can help me in troubleshooting, ????

[Drainy]

well an upgrade wouldn't have deleted files arbitrarily from another instance. I think you might have something else going on here or you've made a mistake somewhere else.

[smolcj]

i tried it, actually i didnt try to upgrade this instance, i had one instance in 5.0.1 and i upgraded it to 5.0.2 and it is working fine.. both have same user names, i think that is the reason it might have deleted some of these folders in this instance