A host reported in the metadata doesn't seem to have sent any event. Why?

[henrit]

I'm not really used to splunk so maybe this question is silly but let's see.

I'm doing the following search, with the time selection set to Today:

| metadata type=hosts index="myindex"

This result is of course a list of hosts with a total_count which I assumed was the number of events from this host for the day.

Then I'm doing (still for today):

index=“myindex” host=“192.168.0.15”

Which returns nothing....

How is it possible that a host appearing in the metadata has in fact no events for the day?

[acharlieh · Accepted solution]

metadata gives information about the index as a whole, it doesn't actually use the selected timeframe as is stated in the docs (empahsis mine):

The metadata command returns data about a specified index or distributed search peer. It returns information such as a list of the hosts, sources, or source types accumulated over time and when the first, last, and most recent event was seen for each value of the specified metadata type. It does not provide a snapshot of an index over a specific timeframe (such as last 7 days).

If you're interested in timeframes with metadata, you may be interested in the metasearch command to start with. This command behaves exactly like search except it only works on the raw data.

For example:

 | metasearch index="myindex" host=“192.168.0.15” | stats count

will tell you how many records have been indexed with host set to that specific host with an event time of today (assuming that's what your time picker is set to) without the overhead of a search going all the way to retrieving the raw data.

I should make a few obscure notes in case they're not obvious 1) searches are based on event time, not index time. So for example if I decided to send a bunch of logs from last week into Splunk today, they would not show up in this search (since the events took place last week). 2) host is simply a key value pair, it has reasonable defaults to being the host sending in data to splunk (host the UF is running on, network inputs may even try to do a reverse DNS lookup), but it can be altered at indexing time. (for example syslog, you typically send to a syslog server, write those to files, and then have Splunk monitor those files. During indexing, Splunk will change the host of the events to the host in each syslog log themselves... if configured to do so)

As you develop Splunk Fu you may start looking at things like the tstats command, which lets you query statistics from index files directly, but I would start with metasearch first (I'm not completely adept with tstats myself yet 🙂 )