Search mechanics understanding

[blurblebot]

Short and sweet:

Why does the search:

bf=1

(no quotes)

take so much longer to run than

"bf=1"

(with quotes?)

Understanding this would probably help me craft my dashboards to be much more efficient overall.

THank you.

[pawan_acharya]

Moreover, for "bf=1" the search engine splits it into
base lispy: [ AND 1 bf ]
This is because of the default segmentation of underscore (_) character in SPL.
This means it checks for "bf" and 1 in events, rather than key value pair. These values could appear in lots of events individually at the first level filter, and hence the search is likely to take lot of time in narrowing down the results.

You could use TERM("bf=1"), which will translate to
base lispy: [ AND bf=1 ]
This would do the exact match of the listed string and hence the results would be way more faster.

Sample example:
index=_internal active_searches
This search has completed and has returned 1,000 results by scanning 88,524 events in 16.991 seconds

base lispy: [ AND active index::_internal searches ]

index=_internal "active_searches"
This search has completed and has returned 1,000 results by scanning 88,668 events in 14.016 seconds

base lispy: [ AND active index::_internal searches ]

index=_internal TERM("active_searches")
This search has completed and has returned 96 results by scanning 96 events in 2.234 seconds

base lispy: [ AND active_searches index::_internal ]

[dwaddle · Accepted solution]

In general, you can use the search job inspector to get an understanding of how Splunk composes searches. In these two specific examples, the job inspector shows two different base searches

For bf=1:

DEBUG: base lispy: [ AND 1 ]

And for "bf=1":

DEBUG: base lispy: [ AND 1 bf ]

When Splunk sees a search term of key=value, it does some optimizations under the assumption that key is an extracted field. As such, it primarily does its search on value. In your first example of bf=1, Splunk searches for everything with the value of 1, and then post-processes that after field extraction looking for a key of bf and a value of 1.

The second example of "key=value" the quotes around it are a hint that there isn't necessarily a field extraction for key, and that key itself is probably a search term within the index.

In short, there are lots of events with "1" that must be post-filtered based on field extraction, but much fewer events with both "bf" and "1"

A good bit of this is based on my limited knowledge of the meaning of the output of the search job inspector -- hopefully someone with more knowledge of the internals of the search engine can elaborate / correct as necessary.