How does Splunk choose events to scan?

[danielrusso1]

How does Splunk determine which events to scan in order to find results?

For example, say I run a query to find a particular System log event code on a particular server (over a 7 day period):

index=main sourcetype="WinEventLog:System" host=server2 EventCode="33"

I get back 8178 results by scanning 8450 events.

How did Splunk choose the 8450 events to scan?

If I search the total number of System log events over the same period (regardless of event code):

index=main sourcetype="WinEventLog:System" host=dtweb2p*

I get back 10,000 results by scanning 14,589 events.

I understand why I received more results obviously. What I do not understand is why they universe on which Splunk searched increased as well. To me it would seem logical that the number of System log events to search on would be the same.

[Ayn · Accepted solution]

It's actually the other way around. Splunk has no concept of fields until very late in the search process (except for index-time fields), so it searches for the values you supply and THEN check to see if any found values are bound to the field you specify. See more in @dwaddle's EXCELLENT answer on this here: http://splunk-base.splunk.com/answers/54207/slow-search-when-evaluating-a-numeric-value?page=1&focus...

So in your first search, you have a number of fields that you use to filter which events Splunk should retrieve. A number of these are already set at index time, so Splunk will filter on both field and value right away. One field, EventCode, is extracted at search-time. What Splunk does is it looks in the index for events having the value "33", and THEN checks if any found values are bound to the field EventCode. So when it says it's scanned 8450 events and found 8178 matching results, the events that were scanned but found not to match are the ones that have the value "33" but not in the EventCode field.

EDIT: As for your second search, I suspect Splunk needs to look at a bunch of events to see which have a matching host field because of your wildcard.

[jtrucks]

I suspect in this case, the first search contains a total population of events that have the field EventCode with any value (even null) and the latter search contains a total population of events both with and without the field EventCode in it.

[danielrusso1]

You would think so, however the EventCode field appears in 100% of the results for the latter search.