How to bulk delete multiple hosts from Splunk that have not phoned home within a set interval?

[brent_weaver]

Good morning. I am brand new to Splunk and so far so good 🙂

We operate in the MS Azure Cloud and many of our systems are Paas Servers. This means when scaling, VM's come up and and are deleted frequently. This has the potential of leaving 100's of "dead" splunk clients in our config requiring me to delete them one by one, almost EVERYDAY :)... So what I am trying to figure out is how to bulk delete machines that have not phoned home in a set interval (i.e. 8 hours for example?) Where are these host definitions kept? I would like to keep the index info but delete the host itself.

Make sense? Any help is MUCH appreciated!

[brent_weaver]

I got an answer.. All I need to do is restart the splunkd service and it will remove all fwd hosts and will pick up those that are "current". I cannot thank you enough for your time on this issue. I have a career because of people like you taking your valuable time to help others!

Thanks a million!

[richgalloway]

Now I understand what you're trying to do. I don't have much experience managing forwarders, but I know the Deployment Monitor app gets its host list from the metrics log. I don't know where the Forwarder Management page get its list.

Have a look at the REST API Reference manual. There are interfaces that allow you to fetch the forwarder list and delete selected entries from the list. It's not a bulk operation, but you should be able to script it.

[brent_weaver]

I took your advice and see what you mean with REST API. Thank you very much for taking time to help me with this!

[richgalloway]

In general, one does not delete data from Splunk. Data ages out over time, but is rarely deleted manually.

How to handle dead machines depends on how you are tracking them. If you get or can get "delete" events from Azure, use those to filter out deleted machines from your reports.

[brent_weaver]

Rich - Thanks for taking the time to respond. I am not looking to delete data, instead remove hosts that no longer exist. I actually want to keep the data in the indexes, just want the host to go away.

Thanks!

[PopcornBob]

Just wanted to say thanks for coming back and posting an answer. Had this issue today, and a splunkd restart on the deployment server worked perfectly.

[richgalloway]

You say you delete hosts yourself each day. How do you do that? Once we understand the manual process we can try to help you automate it.

[brent_weaver]

Rich - I am referring to removing client machines/hosts. This servers that Splunk is "monitoring". I am not sure how else to state it. Splunk is not at all intuitive and I do not even know how to remove a host in the web interface let alone in some backend script. I have searched and searched on the web for documentation on removing hosts and I cannot find anything! My frustration with this product is mounting by the day!

[richgalloway]

Brent, How are you seeing these hosts you wish to remove? If you're using a search, please provide the search. If you're using an app, which one? I'm trying to get a better picture of your environment. Vanilla Splunk only stores data in indexes, which cannot be modified. It doesn't store host information anywhere. Apps, however, may have other storage.

[brent_weaver]

Thank you for your patience! If on the splunk deployment server, you go Settings -> Forwarder management. This view will show all the hosts that are forwarding to Splunk. This is where I can click Delete Record in the Actions col. Many of my hosts show up as not having phoned home in days... These hosts will never phone home again because they are Paas server in Azure, and when rebuilt will get a totally new hostname sid etc...

Make sense? Hopefully I have given you enough information. Again thank you VERY much for taking the time to help me here.