How to change settings on a forwarder via REST API?

[dominiquevocat]

How can I change settings on a forwarder via REST?

Settings I want to be able to modify are:
- deploy.poll frequency
- deploy-poll hosts(s)
- indexer
- search app inputs (stuff added by a local admin on the machine via command line)

Any pointers to the docs would be welcome as well and maybe an example 🙂

[dominiquevocat]

Ended up doing my own custom command and a bit of UI and it is available at https://splunkbase.splunk.com/app/2775/

[jkat54]

Is this a "heavy" or "universal" forwarder? There are fewer options in the UF's API so this is why I'm asking.

Theoretically, if you can use a deployment server to deploy applications to a forwarder, then you can deploy new inputs and settings via REST. However, the way you connect a UF to a deployment server seems to suggest the UF will only ask the deployment server for apps, and so you'd have to establish a handshake and pretend to be a deployment server. As for changing the deployment server (deploy-poll) remotely, you might as well reinstall remotely using SCP, WinRM, PowerShell Remoting, WMI, etc. because it's currently impossible to set it remotely on UF's.

All of the above applies to UF's only. I have a script for changing deployment servers on heavy forwarders if you're interested, here it is... it's pure development stage and is meant to be inside an app called dschanger, under the bin folder. It is meant to be used under windows or linux but was developed and tested on windows only. Use at your own risk!

# dschanger.py is meant to be executed using splunk python context as follows:
# $SPLUNK_HOME/bin/splunk.exe cmd python ..\etc\apps\dschanger\bin\dschanger.py
# $SPLUNK_HOME/bin/splunk cmd python ../etc/apps/dschanger/bin/dschanger.py 
# Or as a scripted input via inputs.conf examples below:
#
#Example Inputs.conf# 
#
##windows input, will be auto-disabled by splunk if application is installed on linux env
#[script://$SPLUNK_HOME\etc\apps\dschanger\bin\dschanger.py]
#index = _internal
#interval = 0 * * * *
#source = dschanger.py
#sourcetype = admin_scripts
#
##linux input, will be auto-disabled by splunk if found on windows env
#[script://$SPLUNK_HOME/etc/apps/dschanger/bin/dschanger.py]
#index = _internal
#interval = 0 * * * *
#source = dschanger.py
#sourcetype = admin_scripts
#
#How to use:
#get a list of your heavy forwarders before and after compare, fix descrepancies
#set the variables below (newDeploymentServerURI, user, password, splunktimeout
#setup new deployment server, start it atleast once, log in to set admin pass
#copy apps from old deployment server
#copy serverclass.conf from old deployment server
#add dschanger app to old deployment server, with this python script and the inputs.conf described above 
#deploy to all servers
#servers will restart multiple times
#monitor logs via 'index=_internal sourcetype=splunk_python dschanger'

newDeploymentServerURI = "127.0.0.2:8089"  #<IP_address/hostname>:<management_port>
user = "admin" #user on forwarders
password = "password" #pass on forwarders
splunktimeout = 60 #amount of time to wait for splunkd to stop & start

# needs test to see if newDeploymentServerURI = local ips, if so err out otherwise splunk wont restart if you give it ip like localhost, 127.0.0.1, etc.

try:
 import splunk.mining.dcutils as dcu
 logger = dcu.getLogger()
 logger.info("New deployment server to be configured as " + newDeploymentServerURI)
 import subprocess, sys, os, time, re
 cred = user + ":" + password
 splunkHome = os.path.normpath(os.environ["SPLUNK_HOME"])
 if os.name == 'nt':
  splunkBin = splunkHome + "\bin\splunk" 
 else:
  splunkBin = splunkHome + "/bin/splunk" 
 if os.path.isfile(splunkBin):
  try:
   currentDeploymentServerMsg = str(subprocess.check_output([splunkBin,'list','deploy-poll','-auth',cred]))
   subprocess.call([splunkBin,'set','deploy-poll',newDeploymentServerURI,'-auth',cred])
   if os.name == 'nt':
    # really didnt want to use sc.exe ... but subprocess.call, os.system, and others caused timeouts on restarts in windows 10 during dev when calling splunk.exe to do the job
    subprocess.call(['sc','stop','splunkd'])
    time.sleep(splunktimeout)
    subprocess.call(['sc','start','splunkd'])
    time.sleep(splunktimeout)
   else:
    subprocess.call([splunkBin,'stop','-auth',cred])
    time.sleep(splunktimeout)
    subprocess.call([splunkBin,'start','-auth',cred])
    time.sleep(splunktimeout)
   currentDeploymentServerMsg = str(subprocess.check_output([splunkBin,'list','deploy-poll','-auth',cred]))
   if newDeploymentServerURI in currentDeploymentServerMsg:
    logger.info("New deployment server has been configured as " + newDeploymentServerURI)
   else:
    logger.error("failed to change the DS to " + newDeploymentServerURI + " this was received instead: " + currentDeploymentServerMsg)
   currentSplunkStatus = str(subprocess.check_output([splunkBin,'status','-auth',cred]))
   if "Running" not in currentSplunkStatus:
    logger.warn("Tried to restart splunkd on host: " + str(subprocess.check_output("hostname")))
   else:
    logger.info("Splunk successfully restarted")
  except Exception as e:
   logger.exception(e)
 else:
  logger.error("Couldnt find file path " + splunkBin)
except Exception as e:
 logger.exception(e)

I also found this neat answer on how to send data in via the REST api on UF's:
https://answers.splunk.com/answers/90466/can-i-call-rest-endpoint-of-universal-forwarder-to-pass-log...

I also noted that people mentioned that you cant curl/browse the UF's API unless you set a different password from the default admin:changme.

As for changing inputs and forwarding to different indexers, that can definitely be done on a Heavy Forwarder. I would suggest you check out the rest api guide for more details on how to do so. http://dev.splunk.com/restapi

I hope something in the above helps!

Cheers,
JKat54

[dominiquevocat]

It is a universal forwarder. Thing is i have usually no means to deploy binaries and settings on server and wish to be able to set purely splunk settings myself. I would like to build it into the splunk app "forwarder query" ( https://splunkbase.splunk.com/app/2775/ ) i.e. i would like to get a number of servers and apply a setting to those from the deployment server as i have access to the rest endpoint of all forwarders from the server where the deployment server runs.

I am ok with deploying an app with a script changing settings locally but i was thinking it should work via rest :-).