Splunk Search

Number of hosts each day?

kwaingrow
Path Finder

I'm looking for a count of the number of hosts each day that have sent events to splunk. I know another department spun up a VERY large number of servers in the last few days and I just don't know how many or which department until I can identify the servers.

My hope is to get the list of hosts in the last 24 hours and get a list for last week, and compare (diff) the 2 lists.

The follow shows the same number of hosts whether it is over 15 minutes or 24 hours on any given day. The number of hosts is always the same.
| metadata type=hosts | table host
or
| metadata type=hosts | stats count by host

The following doesn't show enough servers. We have much more sending to splunk then what this outputs.
index=_internal per_host_thruput | timechart span=1d dc(series) as hosts

Any help is greatly appreciated!

Tags (4)
0 Karma

sideview
SplunkTrust
SplunkTrust

The | metadata command will only ever give you the current stats, so it wont do you any good as far as comparing today's hosts with last week's host.

and the _internal metrics data technically only lists stats from hosts that are significant contributors. So if there are a lot of hosts that are contributing relatively little, they wont show up there at all.

The answer may be to just spin up a search that searches every event.

If you run this search over a given time period:

* | fields host | stats count by host | sort host 

it will give you an exhaustive list of all of the hosts in that period.

If you run this search:

* | fields host | timechart dc(host)    

that will give you a graph of the number of distinct hosts over time, which might be useful.

And if you want to try doing the whole diff of this weeks hosts versus the hosts from a week ago, here are some searches that can do that:

* | fields host | eval daysAgo=(now()-_time)/(24*3600) | eval interval=case(daysAgo<1,"recent",daysAgo>7,"last week") | fillnull interval value="in between" | timechart dc(host) by interval

* | fields host | eval daysAgo=(now()-_time)/(24*3600) | eval interval=case(daysAgo<1,"recent",daysAgo>7,"last week") | stats dc(host) by interval

kwaingrow
Path Finder

No, cause I don't know what hosts are new. That is my problem I am trying to solve. Yes. No errors in splunk logs. I am trying to search the last 24 hours and 24 hours one week ago so I can compare.

0 Karma

gnovak
Builder

Is there any way to check one of these hosts that should be showing up in splunk? Is it forwarding to the right indexer in outputs.conf? Are there any errors in splunkd.log? Are you searching last 24 hours I supposed when looking for hosts?

0 Karma

kwaingrow
Path Finder

Summery page: No, only 10 show up, I have thousands of servers
10 to 100 servers could be spun up at any time. They use an image so when they come online they use existing predefined inputs and sourcetypes. When there is a spike in my license usage it is typically a host or cluster spamming splunk (ie debug is turned or there is a serious problem across a cluster and massive errors are logged). Since there is no abnormality in the events being sent to splunk and I know we are turning up servers within the company, I need to get a handle on this so I know when we will max out our license.

0 Karma

gnovak
Builder

Are the hosts visible on the summary page? Are they at least showing up in splunk? These new hosts that were spun up, were they set up with inputs and to forward data to the splunk indexer? If they haven't sent any data to an indexer they won't show up as a host at all.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...