If you're running 6.5.0, you should disable search_optimization::predicate_merge

[martin_mueller]

In 6.5.0 Splunk added a bunch of search optimizations, see http://conf.splunk.com/files/2016/recordings/optimized-search-optimization.mp4 / http://conf.splunk.com/files/2016/slides/optimized-search-optimization.pdf for details on what was added.

Sadly, one of those optimizations triggers a bug in SPL. The optimizer would turn this

index=_internal | where user="admin"

into this

index=_internal user=CASE("admin")

because where is case sensitive, search is not. However, if there is a calculated field present for user, the second search is turned into this lispy expression:

[ AND index::_internal admin case ]

That will silently return incorrect search results.

If you have a field that's not based on indexed tokens, e.g. extracting part of a word, you might search for it like this:

index=_internal | search my_field="value"

The optimizer will again turn it into this:

index=_internal my_field="value"
[ AND index::_internal value ]

That will again silently return incorrect search results because there is no indexed token value.
Note: Regardless of the optimizer changing the results, you should usually use fields.conf to properly declare "this field is not an indexed token", then the optimized results will be correct again and you won't have to use the piped search yourself.

Hence if you're running 6.5.0 already, you should disable that part of the optimizer in limits.conf:

[search_optimization::predicate_merge]
enabled = false

Setting that will give you correct results again.
Case references for Splunkers: 408188, 408195

[martin_mueller]

The first bug (calculated fields) has been fixed in 6.5.1, it seems. IMHO you can now use predicate_merge fairly safely.
While the second bug (field not based on indexed tokens) is still there, it's much harder to trigger and works well together with the noop workaround... plus, you should just define fields.conf 🙂

[Lucas_K]

Where should this change be made in a distributed environment?

Seems to be a search time option so just search heads?

[jrodman]

The search-optimizer runs at the initial comprehension of the search string, so should only be relevant or be used on a search head. As always it may be useful to bring search config to indexers just for the case of troubleshooting by launching searches directly on an indexer, but that's very much an edge-case scenario.

[martin_mueller]

Indeed... to be answered by a release 😄

You can also selectively enable the optimizer by passing true to noop even if the global setting is disabled.

[jrodman]

This is a difficult one to answer 😜

I should mention that if for some reason you want to disable on a search-by-search basis, you can by adding a command to the search |noop search_optimization=false