How to search all hosts that have been a part of a project that I do and do not see logs from?

[hegga]

We've now set up and installed our Splunk instance, gotten data into it, and are soon ready to close the project. But before we do so, I'd like to do a QA where we check if we are getting logs from all of the hosts that have been a part of the project. I could just do a:

index=* | stats count by host | table host, count

But this would only give me a list of all of the hosts that I see logs from, I would also like to add the hosts (if any) that I don't see logs from in this table with the count of zero. How can I achieve this?

I don't have my list of hosts in a lookup table or anything, so I will have to specify them in the query. Can anyone help with this?

[maciep]

Like others have alluded to here, you need to elaborate on the scenario you're trying to account for. If you're concerned that you may have completely missed installing a forwarder on some of the servers needed for the project, then you need to have a list of expected servers somewhere to create a lookup from. Unless your servers are named in a way that you can logically determine which ones should be there, e.g. server00, 01, 02 ....37, splunk isn't quite omniscient yet 🙂

If you're concerned that of the forwarders you installed that maybe some are configured right or not sending the project logs, then I think some of the other suggestions here will work.

[hegga]

Thank you for your reply!

My main scenario is to identify potential servers were the Splunk has not been installed at all. I have a list which contains all of the servers were my service provider should have installed the agent, but before I close the project
I want to make sure that they really have installed it on all of the servers. As stated earlier I have a list of all of the hostnames which can be used in a query, but I need help in order to identify all the servers which does not send any logs to Splunk at all.

[maciep]

Not sure if you tried Ryan's search,but you could create a lookup in splunk with your list of servers - just a simple csv file with one column and header named host, for example. Call it servers.csv?

Then you could run a search to join that list with servers splunk knows about. Something like this (untested)

| inputlookup servers.csv | join type=left host [|metadata type=hosts ] | where  isnull(lastTime)

So the inpulookup command will essentially return your csv file, which is just a list of hosts. And the metadata command will return some basic info about all of the hosts splunk knows about, including the last time it saw something from a host.

If you do a left join, then you'll keep everything from your lookup and only those items from metadata that match. So if any of the servers in the list don't have the fields that come with the metadata command, like lastTime, then they may have been missed.

[ryanoconnor]

@hegga see if you can try out my search above exactly as it is. You should be able to modify it to use your specific host names and find all hosts not sending data.

[ryanoconnor]

I of course would recommend the lookup table route, but If you're looking to sort of "create" hosts that don't actually exist, you could also do something like this:

index=* | eval host=host.",test_host, test_host2" | makemv host delim="," | mvexpand host | dedup host | table host

This would give you a list of hosts that actually exist, with the addition of two "fake" hosts called test_host and test_host2.

From there you could use the join command on the host field to actually join your list of hosts to actual data in a subsearch:

index=* | eval host=host.",test_host, test_host2" | makemv host delim="," | mvexpand host | dedup host | table host | join type=outer host [search index=* | table host index]

Finally, you could filter by any hosts with no data in any index

index=* | eval host=host.",test_host, test_host2" | makemv host delim="," | mvexpand host | dedup host | table host | join type=outer host [search index=* | table host index] | fillnull index value=NO_DATA | search index=NO_DATA

[hegga]

Hi,

Thank you for your replies! I've tried the searches above, but they are very, very slow so my search jobs are expiring before they finish.

[ryanoconnor]

Ya that's where a lookup table would ultimately make this a lot more efficient and why it's probably going to keep getting recommended 🙂

What time period are you running my searches over?

[somesoni2]

Every host will send two types of data, their internal logs (comes by default if forwarder is set correctly) and custom monitored data (this is what you deployed onto forwarders). If a host is not sending both type of data, then it'll be nearly impossible to get list of those. If they are sending internal logs (at least), then you can run a query to confirm if they are sending custom monitored data. Something on the line of this (will list hosts which are not sending custom monitored data, but sending _internal data)

| tstats dc(index) as indexcount WHERE index=* OR index=_internal by host | where indexcount =1