Results returned by a search not matching up with events

[responsys_cm]

We're in the process of testing a number of different types of data to properly size the expansion of our Splunk license. One data set we're looking at is generated by the tools from the Argus network monitoring project.

I've indexed the flow records and performed searches like: index=si-netflow | head 500,000

The search results will show 500,000 results, but if I click on "i" and see the search properties, only 1000+ events are returned. And if I tab through the results pages, they end at the 1000+ event.

Here are the search statistics:

Search job inspector

This search has completed and has returned 1,342 results by scanning 504,571 events in 42.822 seconds.

The following messages were returned by the search subsystem:

DEBUG: base lispy: [ AND index::si-netflow ]

DEBUG: search context: user="admin", app="search", bs-pathname="/opt/splunk/etc"

(SID: 1363321056.11)

Execution costs

Duration (seconds) Component Invocations Input count Output count

0.046 command.fields 46 504,571 504,571

0.046 command.head 46 500,000 500,000

0.071 command.prehead 46 504,571 500,000

25.463 command.search 46 - 504,571

0.215 command.search.index 45 - -

0.045 command.search.calcfields 45 504,571 504,571

0.045 command.search.fieldalias 45 504,571 504,571

0 command.search.index.usec_1_8 15 - -

0 command.search.index.usec_64_512 1 - -

0 command.search.index.usec_8_64 85 - -

12.549 command.search.rawdata 45 - -

8.945 command.search.typer 46 504,571 504,571

2.852 command.search.kv 45 - -

0.046 command.search.tags 46 504,571 504,571

0.045 command.search.lookups 45 504,571 504,571

0.02 command.search.summary 46 - -

0.013 dispatch.check_disk_usage 5 - -

0.11 dispatch.createProviderQueue 1 - -

0.166 dispatch.evaluate 1 - -

0.165 dispatch.evaluate.search 1 - -

0.001 dispatch.evaluate.head 1 - -

25.504 dispatch.fetch 46 - -

0.03 dispatch.preview 30 - -

0.02 dispatch.readEventsInResults 1 - -

0.806 dispatch.results_combiner 46 - -

25.501 dispatch.stream.local 46 - -

12.916 dispatch.timeline 46 - -

0.093 dispatch.writeStatus 35 - -

0.066 startup.handoff 1 - -

Why does such a broad search not return the full 500,000 events?

Thx.

Craig

[hexx]

I'm fairly sure that what you are observing is a somewhat undesirable interaction between the head search operator and the max_events_per_bucket parameter in limits.conf:

max_events_per_bucket = <integer>
* For searches with status_buckets>0 this will limit the number of events retrieved per timeline bucket.
* Defaults to 1000 in code.  

First of all, let's explain what that parameter does:

When you build a non-reporting search (i.e: a search that matches and returns events but does not perform any statistical aggregation on them), Splunk assumes that you are either looking for a needle in a haystack or that you are exploring your data.

It also reasons that in the latter scenario, if your search returns a very large number of events (say, many thousands or more) you don't really want to manually browse through all of them, and therefore that the cost in network traffic, CPU, disk and memory usage of actually transferring all of those events from the indexers to your search-head is not worth it.

So, what Splunk does instead is that it fetches up to 1,000 (or whatever non-default value max_events_per_bucket is set to) for each time slice that the search view timeline visualization renders, thus allowing you to explore up to 1,000 buckets for each of these time buckets in the UI.

My suspicion is that head interacts with this parameter somehow in a way that results in the fetching of only one time bucket, which limits the number of events fetched to 1,000.

Note that head is not a very search-efficient operator as it requires matched events to be returned to the search-head before sorting (in search order) can be performed there. Typically, it is used for casual data exploration and to limit events returned to a usually small number.

So, depending on what you actually want to do, you may want to:
* Drop head and limit the number of events returned using a time range.
* Increase max_events_per_bucket if you want to be able to browse more events in the search UI. Do note that this comes at a performance cost for the search-head.

That being said, this behavior seems rather counter-intuitive as reported and possibly indicative of a bug with the head search operator. As such, I encourage you to open a case with Splunk Support to report this issue.