Getting Data In

How to get a scripted input to do something and then restart Splunk?

fairje
Communicator

So I am working on handling variable situations in a deployed environment so the way I was solving this issue was to use Splunk's scripted input capability to run an initial script (powershell) on the host, find some data about the host, update my "ClientName" field in the deploymentclient.conf, and then restart Splunk.

Everything is going great up until the "restart Splunk" requirement. Which must happen because otherwise it won't update the ClientName until something else causes Splunk to restart.

In powershell you can call a "Restart-Service" command which will restart the service. The problem is, Splunk is what executed the underlying script, so I am essentially telling it to restart itself. This is a problem, because it then kills the script when Splunk stops, and nothing controls it to turn it back on again.

So I thought I would look into spawning a background process through powershell to force it to be an independant process and it seemed like it going to work, until it stopped in the same place... and my background task closed out as well only I can't see what happened because it is in the background...

The relevant part of my powershell script is below:

Start-Process powershell -ArgumentList "-NonInteractive -command & ({Restart-Service SplunkForwarder})"

I checked this command does work by running it manually on the host. It pops up a new powershell window and you can see it restart the service. And I even tried modifying the command to instead add the "-NoExit" parameter. This time, it runs, stops the splunk service and the powershell.exe stays alive... but it never turns splunk back on again...

In the splunkd.log I still get the same:

01-28-2016 18:57:57.234 -0500 WARN  ProcessRunner - Process with pid 8900 did not exit within a given grace period after being signaled to exit. Will have to forcibly terminate.

Anyone else try using Splunk Scripts to make a change to your Splunk environment and then kick a restart? If so how did you handle it?

Edit: Below is additional parts of the Powershell to show more the line of thinking I was going down:

if (Test-Path $localDC) {
    #local/deploymentclient.conf exists already
    $localclientsetting = (Get-Content $localDC | Select-String -Pattern "clientName") -replace "clientName = ", ""
    if ($localclientsetting -ne $ClientName) {
        #local/deploymentclient.conf doesn't match with new Client Name
        #host has moved, replace with new ClientName
        (Get-Content $defaultDC) -replace "\[deployment-client\](\r|\n)*","[deployment-client]`r`nclientName = $clientName" | Set-Content $localDC
        #Restart-Service SplunkForwarder -WarningAction SilentlyContinue
        Start-Process powershell -ArgumentList "-NonInteractive -command & ({Restart-Service SplunkForwarder})"
    }
} else {
    #first time deployment, create the file
    (Get-Content $defaultDC) -replace "\[deployment-client\](\r|\n)*","[deployment-client]`r`nclientName = $clientName" | Set-Content $localDC
    #Restart-Service SplunkForwarder -WarningAction SilentlyContinue
    Start-Process powershell -ArgumentList "-NonInteractive -command & ({Restart-Service SplunkForwarder})"
}
0 Karma
1 Solution

gtriSplunk
Path Finder

Hi fairje,

On Linux you can fork a non-child process to restart Splunk, but since you are using Windows take a look below.

When Splunk starts up it will enable all scripted inputs which will cause them to run. If your scripted input restarts Splunk you will get stuck in a loop. One solution is to use an if-statement to trigger the restart. Let's say your scripted input changes the hostname to "mynewhostname", you can do something like this:

if hostname != mynewhostname:
change it
restart splunk

This would only restart Splunk once and eliminate the restart loop, assuming your hostname doesn't keep changing.

If you are using a deployment server you can delete the entire app directory which would cause the deployment server to download the app again and restart Splunk (assuming you have the app triggered to do so). Take a look at this app for an exact example:
https://splunkbase.splunk.com/app/2722/

GTRI Splunk Team!

View solution in original post

minbzk
New Member

Took a while to figure out how to restart the UF from a scripted input. I tried many options, in most cases the forwarder stopped but never started again. Not an option. The root cause is that a stop of splunkd on Windows forces the parent process of the powershell script itself to stop executing. This solution works (tested with UF version 6.3.3 an 6.4):

I used a .path file, a powershell script (.ps1) and a command file (.cmd). Put all three scripts in the same app/bin folder.
The path file enables bypassing the default security Powershell security policy, this (normally) prevents the .ps1 from executing when called directly from inputs.conf. The .ps1 is the do something app, the cmd file is necessary to spawn a separate process (not a child of splunkd) that will restart the forwarder.

in YOUR_APP\default\inputs.conf on the forwarder:

[script://.\bin\<myscript.path>]
interval = -1
disabled = 0

myscript.path:

$SystsemRoot\System32\WindowsPowershell\v1.0\powershell.exe -ExecutionPolicy Bypass -command " &'$SPLUNK_HOME\etc\apps\YOUR_APP\bin\<myscript.ps1>' "

The file myscript.ps1 code to disable script input in inputs.conf
You really need this code to prevent boot loops. Do not put other stuff in this inputs.conf, without modifing this code. It will replace every = 0 to = 1 present in the inputs.conf. You might place other inputs.conf stanza's in a different inputs.conf.

$scriptdir = Split_path $script:MyInvocation.MyCommand.Path
$inputsconf_file = resolve-path ($script_dir, "..", "default", -join "\")
$inputsconf_file = $inputsconf_file, "inputs.conf" -join "\"
 (Get-Content $inputsconf_file) |
    Foreach-Object {$_ -replace " 0", " 1"} |
    Foreach-Object {$_ -replace "=0 ", "=1"} |
    Foreach-Object {$_ -replace "false ", "true"} |
 Out-File $inputsconf_file

The file myscript.ps1 code to initiate a restart:

$reset_script = $script_dir, "ufrestart.cmd" -join "\"
$args = "SplunkForwarder" 
start-process -FilePath $reset_script -Arguments $args

the ufrestart.cmd:

@echo off
sleep 5
net stop %1
net start %1
exit

Have fun!

0 Karma

gtriSplunk
Path Finder

Hi fairje,

On Linux you can fork a non-child process to restart Splunk, but since you are using Windows take a look below.

When Splunk starts up it will enable all scripted inputs which will cause them to run. If your scripted input restarts Splunk you will get stuck in a loop. One solution is to use an if-statement to trigger the restart. Let's say your scripted input changes the hostname to "mynewhostname", you can do something like this:

if hostname != mynewhostname:
change it
restart splunk

This would only restart Splunk once and eliminate the restart loop, assuming your hostname doesn't keep changing.

If you are using a deployment server you can delete the entire app directory which would cause the deployment server to download the app again and restart Splunk (assuming you have the app triggered to do so). Take a look at this app for an exact example:
https://splunkbase.splunk.com/app/2722/

GTRI Splunk Team!

fairje
Communicator

This totally worked. I now can programmatically set my ClientName on all my systems. There were a couple which didn't work, that are likely an error on my side somewhere, but the technique is solid and solves the issue of how to do this.

I basically already had a "Config_Base" app that has my deploymentclient.conf file in it. I just moved the scripting stuff into a new app call "Script_Base_Windows", which I will use for this and any other scripts I decide to make available to all windows systems on the network. The script changes the Config_Base app, and then removes the script app from the system, then when it checks in next time it will redownload the app and restart!

Thanks a lot for the assist here it was greatly appreciated!

0 Karma

fairje
Communicator

Oh wow! I didn't realize someone had made a password reset app!

So based on this, what I might have to do is essentially have two apps. One that has the script in it, and one that has the configuration files in it that I want to change. This way the script app just deletes itself forcing a reload once it drops back down onto the system from the deployment server, right?

Using the logic added to the question above (which I had already worked out ahead of time to avoid a restart loop that I knew would happen since the script is set to run once per restart), we should be good! I'll test this and assuming it works will mark this as the answer!

Edit:
Oh, for those who don't want to click through, download the app, and tear it apart. The magic from this app is in two scripts (one for linux bash and one for windows batch):

BATCH:::

"%SPLUNK_HOME%\bin\splunk.exe" edit user admin -password !_RndAlphaNum! -auth admin:changeme
echo "pwchange ran!"
@echo on
del /F /Q /S "%SPLUNK_HOME%\etc\apps\TA_PWChange"

SHELL SCRIPT:::

$SPLUNK_HOME/bin/splunk edit user admin -password `head -c 500 /dev/urandom | sha256sum | base64 | head -c 16 ; echo` -auth admin:changeme
echo "pwchange ran on `hostname`!"
rm -frv $SPLUNK_HOME/etc/apps/TA_PWChange

So essentially what needs added into my powershell is:
Remove-Item "$SPLUNKAPPS\myscriptapp" -recurse

Note that I defined that variable myself... Although I will be running 6.3 soon-ish on my forwarders, which should allow running powershell in the context of Splunk with the Environment Variables passed into it, currently I am having splunk use a path file to run powershell from my windows directory and pass into it my script.

0 Karma

gtriSplunk
Path Finder

Not exactly, close though. You still can use one TA. If you make the "deploymentclient.conf" that you want installed on the forwarder, save it in the default or local folder of the TA and have the script compare the already installed and "new" conf files. If they are different, replace, and delete the TA folder. If they are the same, do nothing. This IF-statement is critical. Once the TA is deleted the deployment server will push out the TA again, trigger a restart, the script will run and see the files are the same, skipping the delete function.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi fairje,

check out this answer https://answers.splunk.com/answers/129614/how-can-i-restart-all-my-forwarders.html which provides two options to do this or this answer https://answers.splunk.com/answers/91160/restart-a-uf-via-cli-other-remote-means.html uses remote REST calls, which could be scripted as well.

Hope this helps ...

cheers, MuS

fairje
Communicator

I appreciate the response, unfortunately I was trying to avoid having to trigger a restart from outside of host. Because the change is being made to the forwarder, and I don't want to have to keep track of which forwarders have been changed and which ones haven't in order to know when to restart them, it would be better if they could manage this on their own.

In this case, I am manipulating the ClientName field inside the DeploymentClient.conf. Systems that already have this set correctly don't need to do anything. Systems that either need it changed (because the host moves locations), or don't have it set at all (new workstation or server), need to have this updated and restarted.

0 Karma

gtriSplunk
Path Finder

You would also have to set the password on all the forwarder to allow rest calls.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...