Moved Splunk server to a new domain - no host name or ip change

[cjdesrocher]

I moved my splunk server running Windows Server 2008 R2 to a new domain. I initially installed Splunk using the Local System account to run the services. The IP and hostname did not change. Splunk seemed fine. I could get to the webpage, log in and see all my data. Yesterday, I tried to update it from 5.0.2 to 5.0.4 and get these 1 of these 2 errors: "Please make sure that the user running the program has the correct privileges, including being able to create windows services" or "Splunk Installer was unable to launch Splunk First Time run error code 1". I am a domain admin. I noticed that in Programs and Features that Splunk does not show up as an installed program. I added Domain Admins to Full control of D:/Splunk where splunk is installed but still get the errors and still do not see Splunk in Program and Features. I am trying to install it via the cmd window with admin privileges, type msiexec -i splunk.5.0.4.msi but still fails with one of the errors above.

[cjdesrocher]

I ended up changing domains back to my original domain where I could see Splunk in Programs and Features. Before I uninstalled it, I backed up etc, var/lib, and bin/script. However in Windows, some folders are locked. Be sure to check the size of the backup copies with the originals. I had to manually find which folders were missing files. Once the folders were backed up, I uninstalled Splunk, and changed domains back.

Before I installed Splunk on D drive I gave Domain Admins (a small group) full control of the entire drive. I setup a Splunk account in AD with membership in Domain Admins. I wanted to avoid the issue of only the installer being able to see the program and being able to update it. I used the Splunk account to log in to the server, install Splunk and restore the backup files.
Like before with the copy, folders are locked. The Windows copy process took 3 times to get most of the data. I had to manually find which folders were missing files and add files back. I started up Splunk and logged in and success…almost. I could see logs coming in from the agents, search, verified my licenses and apps. Everything checked out, but I had two errors:

1. Can not enable audit.db. Save checkpointstr: unable to open checkpointfile=’…\wineventlog\application’ for write: Access Denied
2. Received event for unconfigured/disabled/delted index=’ _audit’ with source=’source::audittrail’ host=host::hostname sourcetype=’sourcetype::audittrail’ (1 missing)

There were two indexes that were disabled. One enabled fine, the other _audit did not.
Looking at the splunkd.log, I noticed errors dealing with folder discrepancies hot_v1_0 and hot_v1_##. I looked in my backup folder var\lib\Splunk\audit\db and only saw hot_v1_##. In the same folder on D drive there were two folders. I copied the hot_v1_0 for backup and then deleted it. Error 1 cleared but error 2 was still present. I restarted Splunk (from within Splunk) and both errors were cleared. Splunk is back and fully restored.

Also, Splunk now shows up Program and Features for all Domain Admins. I am guessing it because I gave Domain Admins full access prior doing the install. But I am not sure.

[cjdesrocher]

I even created a local administrator account and logged in locally. Splunk still does not show up in Programs and Features. I still can not install the update. I get the error "Splunk Installer was unable to launch Splunk First Time run error code 1"

[cjdesrocher]

I don't have that run as option. I modified 3 UACs:

Behavior of the elevation prompt for Admins in Admin Approval Mode -> Elevate w/o prompting

Detect application installations and prompt for elevation -> Disabled

Run all admin in Admin approval mode -> Disabled

gpupdate /force and my right click menu still does not show run as admin. My .exe files all show run as administrator but not the msi files.

[lukejadamec]

Have you tried running it "as administrator"? 2008R2 is a very untrusting OS.