Substitute alias for linux hosthame

[jamesoconnell]

Splunk searches return hostname by default -- where hostname is the value returned the linux command hostname.
We have a partner whose VMs get refreshed frequently where the underlying ip changes, but the alias remains constant.

What is the best way to have Splunk report the primary alias rather than the linux hostname?

Regards.

[sloshburch]

Also, would you clarify if the verb "refresh" means that the host is rebuilt or just reassigned an IP (like how AWS EC2s do).

[jamesoconnell]

Thanks Burch for the question -- see above.

-Jim O.

[lguinn2 · Accepted solution]

When you install Splunk and then run it the first time, Splunk captures the hostname of the machine and stores it in SPLUNK_HOME/etc/system/local/inputs.conf in the default stanza.
This is where Splunk gets the default host name for any input; but each input can override that by specifying a host name.
The default is set at first start, and after that, it will not be changed by Splunk.

But you are saying that the name changes - how is the host set for the inputs that are collected by Splunk?

I think that you should create a script to set the default host in SPLUNK_HOME/etc/system/local/inputs.conf
You can set it to whatever you like, and just don't override that default for the inputs that you create.

[jamesoconnell]

Interesting solution; I like it.

I didn't know that SPLUNK_HOME/etc/system/local/inputs.conf was the source for hostname field in Splunk searches.

We could set this value to be the alias for the server instance. A complication here is that the splunk forwarder is wiped out each time re-hydration occurs. The forwarder re-installation process would need to specify the constant alias for each cycle.

thank you lguinn.
-Jim O.

[sloshburch]

You could have the system/local/inputs.conf programmatically entered by your reinstatement process. The entry in that file is only generated if it's absent. See what we mean by unpacking a fresh install (tgz only since other formats too easily start splunk as part of install). Before you start splunk, you can set some config, like this, and see how when you start for the first time, it remains.

[jamesoconnell]

Yes, that is what I was planning to do -- have the forwarder install script set the required value in SPLUNK_HOME/etc/system/local/inputs.conf via an input parameter on the install script.

[niketn]

Is VM alias to IP address mapping stored in any existing log being indexed to Splunk?
If not you will have to created lookup table or KV Store to map the same.

[jamesoconnell]

Thanks for the question -- no, there isn't currently a lookup table mapping alias to ip.
Creating one is an interesting option -- it isn't a static relationship is the complexity of this situation.

Every 2 months the underlying host ip for an alias is destroyed; a new cloud instance created, and the residing apps re-deployed in an automated fashion; our platform team are using the term re-hydrate.

The mapping would need to be pulled after every re-hydration cycle from the network CNAME data.

Feels messy.

[sloshburch]

Does the application team that owns the thing running on that host get notified of the new hostname? If it's like that then I would encourage keeping the dynamically changing hostname so the end users can intuitively correlate the host back to the source. Also maybe the utility for notifying (or DNS system) can be splunked and used for such correlation

[jamesoconnell]

Hmm, I thought of indexing the DNS data to build a lookup/mapping of alias to ip.

While it is possible, I am going to go with my original comment and say I think it is messy -- there are potential multiple alias' per ip etc...