Number of hosts each day?

[kwaingrow]

I'm looking for a count of the number of hosts each day that have sent events to splunk. I know another department spun up a VERY large number of servers in the last few days and I just don't know how many or which department until I can identify the servers.

My hope is to get the list of hosts in the last 24 hours and get a list for last week, and compare (diff) the 2 lists.

The follow shows the same number of hosts whether it is over 15 minutes or 24 hours on any given day. The number of hosts is always the same.
| metadata type=hosts | table host
or
| metadata type=hosts | stats count by host

The following doesn't show enough servers. We have much more sending to splunk then what this outputs.
index=_internal per_host_thruput | timechart span=1d dc(series) as hosts

Any help is greatly appreciated!

[sideview]

The | metadata command will only ever give you the current stats, so it wont do you any good as far as comparing today's hosts with last week's host.

and the _internal metrics data technically only lists stats from hosts that are significant contributors. So if there are a lot of hosts that are contributing relatively little, they wont show up there at all.

The answer may be to just spin up a search that searches every event.

If you run this search over a given time period:

* | fields host | stats count by host | sort host 

it will give you an exhaustive list of all of the hosts in that period.

If you run this search:

* | fields host | timechart dc(host)    

that will give you a graph of the number of distinct hosts over time, which might be useful.

And if you want to try doing the whole diff of this weeks hosts versus the hosts from a week ago, here are some searches that can do that:

* | fields host | eval daysAgo=(now()-_time)/(24*3600) | eval interval=case(daysAgo<1,"recent",daysAgo>7,"last week") | fillnull interval value="in between" | timechart dc(host) by interval

* | fields host | eval daysAgo=(now()-_time)/(24*3600) | eval interval=case(daysAgo<1,"recent",daysAgo>7,"last week") | stats dc(host) by interval

[kwaingrow]

No, cause I don't know what hosts are new. That is my problem I am trying to solve. Yes. No errors in splunk logs. I am trying to search the last 24 hours and 24 hours one week ago so I can compare.

[gnovak]

Is there any way to check one of these hosts that should be showing up in splunk? Is it forwarding to the right indexer in outputs.conf? Are there any errors in splunkd.log? Are you searching last 24 hours I supposed when looking for hosts?

[kwaingrow]

Summery page: No, only 10 show up, I have thousands of servers
10 to 100 servers could be spun up at any time. They use an image so when they come online they use existing predefined inputs and sourcetypes. When there is a spike in my license usage it is typically a host or cluster spamming splunk (ie debug is turned or there is a serious problem across a cluster and massive errors are logged). Since there is no abnormality in the events being sent to splunk and I know we are turning up servers within the company, I need to get a handle on this so I know when we will max out our license.

[gnovak]

Are the hosts visible on the summary page? Are they at least showing up in splunk? These new hosts that were spun up, were they set up with inputs and to forward data to the splunk indexer? If they haven't sent any data to an indexer they won't show up as a host at all.