Is there away to reload the value of the servername key in etc/system/local/server.conf ?

[Hemnaath]

Hi All Currently we are facing an issue for Some of the universal forwarders have had their hostname updated, but it is not reflecting correctly in Splunk. That is the value of serverName maps to their original hostname, but not the current hostname.

Is there a way to reload the value of the servername key in /splunkuniversalforwarder /etc/system/local/server.conf

We have deployment server to manage the app configuration, so is there way to can do it from deployment server? I am not sure whether we can manage system/local/.

UF agent version 6.6.1 & 6.6.2

Kindly guide me on this as we could see lots of agents having this issue.

[sbbadri]

@Hemnaath,

you can set/modify servername from command line by issuing below command. so that it will update in $SPLUNK_HOME$/etc/system/local/server.conf

$SPLUNK_HOME$/bin/splunk set newservername.

i hope it will help you.

[Hemnaath]

Hi Sbbadri, thanks for you effort, so we need to login to the remote agent node and execute the below command right. Most of the UF host are in Windows environment.

so what will be the command we need to use in Windows environment.

c:\programfiles \splunkuniversalforwarder\bin

splunk set newservername.

is this right.

[sbbadri]

yes, you are right c:\programfiles\splunkuniversalforwarder\bin> splunk set newservername

[Hemnaath]

thanks sbbadri, I have final think to ask before executing the above command, we have a deployment server where we manage the app configuration and so is there a way we can push it from deployment server .

suppose if we are logging into the individual remote agent one by one and execute the above command and at the same time when we execute splunk reload deploy-server command from deployment server will it change the server.conf .

Kindly guide me on this .

[sbbadri]

You can do it two ways,

1) Write a bat scripts and update all the remote servers.
2) By creating app in Deployment server. App should contain scripted inputs and script should login to each remote server, take a backup of existing server.conf and implemented cli command to update server.conf. Once implementation is done you need remove the app from deployment server. So that it won't re-apply again and again.

Scripts is necessay because you have multiple servers and each serves have different servername.

[Hemnaath]

Thanks alot sbbadri, I will try to test in one of the server by executing the command manually and check whether servername key is getting updated with the right host name.

[sbbadri]

if it solves your problem, don't forget to vote or accept the answer

[Hemnaath]

Hi Sbbadri, thanks for your effort, hey i got another issue now, i need to filter out which are the UF had their hostname updated, but it is not reflecting correctly in Splunk.

is there a way in splunk ?

thanks in advance.

[sbbadri]

Right now this solution is coming on top my mind.

1) By using splunk query save the list of host in a lookup before changing the host name.
2) Again using splunk query save the list of host in a lookup after update the host name or query the current data against previously saved lookup.

so that you have old list of severs and new list of servers and their update as well.

[Hemnaath]

Hi sbbadri, thanks for your effort on this, hey is it possible to share the information ? from which part of the world you are based out of.

[sbbadri]

use this below query on DS

1) | inputlookup dmc_forwarder_assets where status="missing" AND os=Windows* | rename hostname as before_hostname | fields forwarder_type before_hostname os status version | outputcsv hostname_before_update_servername
so the hostname which currently missing will be saved to this hostname_before_update_servername

Note: if outputcsv is not working try with outputlookup

2) Update the servername using cli command on remote server
3) | inputlookup dmc_forwarder_assets where status=missing AND os=windows* | lookup hostname_before_update_servername before_hostname as hostname OUTPUT forwarder_type before_hostname os status version | table hostname forwarder_type before_hostname os status version

do it for one server. If the output is good. Then go ahead with rest of the stuff.

4) On search head : index=_internal host=before_hostname OR host=hostname. you should get results.

I hope this helps you.

[Hemnaath]

thanks sbbadri, when I execute the dmc_forwarder_asset I could see these details in the lookup table

arch, avg_tcp_eps, avg_tcp_kbps,avg_tcp_kbps_sparkline,forwarder_type,guid,hostname,last_connected,os,status, sum_kb,version

But the moment when it execute next set of search term "where status="missing" AND os=Windows*" data's under this column are left blank.

when ran next set of search term "| rename hostname as before_hostname | fields forwarder_type before_hostname os status version " in before_hostname there is no data its left blank.

and sure why this search is used | outputcsv hostname_before_update_servername

Kindly guide me sbbadri.

[sbbadri]

| outputcsv hostname_before_update_servername is used to store missing forwarder host name list to a csv(lookup) file. so that once you change the servername for those missing host, it will come under active server. so that you can confirm that you have updated the missing forwarder correctly.

[Hemnaath]

Hi sbbadri, I have doubt doing scripted inputs in splunk, it is the first time i am going to use the scripted input. Currently we have an app called XXX-IA-win and under bin I could see two batch file "win_installed_apps.bat" & "win_listening_ports.bat" already present in to it, and these batch files are called in the inputs.conf file like this.

Scripted Input (See also wmi.conf)

[script://.\bin\win_listening_ports.bat]
disabled = 0

Run once per hour

interval = 3600
sourcetype = Script:ListeningPorts
index = win

[script://.\bin\win_installed_apps.bat]
disabled = 0

Run once per day

interval = 86400
sourcetype = Script:InstalledApps
index = win

And also I could see another folder under the defaults-->bin-->ta_windows --->models --> I could see some init.py,init.py0, input_.py & input_.py0 -- Not sure what it does ? so do i need to do anything here.

Question

1) Shall I create new batch file called servername_keyvalue_update.bat and I shoud call this bat file in the inputs.conf stanza like this

[script://.\bin\servername_keyvalue_update.bat]
index = win
sourcetype = Script:servername_keyvalue_update
disabled = 0

then it can be deployed via deployment server to the remote host.

similarly we need to create a shell script to update the server name in UNIX related OS right.

thanks in advance.

[sbbadri]

Please go through below link,

https://docs.splunk.com/Documentation/SplunkCloud/6.6.1/AdvancedDev/ScriptSetup

[Hemnaath]

Hi Sbbadri, I had gone through the link but not sure whether we can use the scripted input concept for this purpose. So is it possible to use the scripted inputs. I am really confused now. sbbadri can you please help me on this.

[Hemnaath]

hi sbbadri, can u guide me on this.