Why is search command TERM not working in one system but does in another?

[sansay]

We just found out that the search command TERM does NOT work when used on extracted fields in one of our Splunk Enterprise environments. But it does in another. The 2 systems have some different configurations and handle a different amount of data, but they are both Enterprise and as such, I would expect search commands to work the same way.

For instance:
index=_internal host=indexer TERM(group=thruput) TERM(name=thruput)
will return nothing in system 1 while it returns the correct events in system 2.

However using CASE works in both:
index=_internal host=indexer CASE(group=thruput) CASE(name=thruput)
returns the correct events in both system 1 and system 2

1. Why does TERM work in system 2 but not in system 1
2. What method can I use to identify the root cause
3. Why do we need to use TERM or CASE if the fields are extracted? I thought that looking for group=thruput name=thruput would work faster since these are extracted, and therefore indexed, fields.

The following section shows the 3 main queries I ran over a 24 hours time range and results.

SYSTEM 1 (AWS)

earliest=5/2/2019:0:0:0 latest=5/3/2019:0:0:0 index=_internal host=indexer TERM(group=thruput) TERM(name=thruput)
0 events

earliest=5/2/2019:0:0:0 latest=5/3/2019:0:0:0 index=_internal host=idx CASE(group=thruput) CASE(name=thruput)
found: 354,372 events
runtime: 18.818 seconds

earliest=5/2/2019:0:0:0 latest=5/3/2019:0:0:0 index=_internal host=idx group=thruput name=thruput
found: 351,096 events
runtime:28.892 seconds

SYSTEM 2 (Data Center)

earliest=5/2/2019:0:0:0 latest=5/3/2019:0:0:0 index=_internal host=indexer TERM(group=thruput) TERM(name=thruput)
found: 86,893 events
runtime: 2.183 seconds

earliest=5/2/2019:0:0:0 latest=5/3/2019:0:0:0 index=_internal host=indexer CASE(group=thruput) CASE(name=thruput)
found: 86,893 events
runtime: 2.187 seconds

earliest=5/2/2019:0:0:0 latest=5/3/2019:0:0:0 index=_internal host=indexer group=thruput name=thruput
found: 86,397 events
runtime: 6.2 seconds

[maciep]

I'd probably start with segmenters.conf to be sure they are the same between systems. That's the file that defines the rules for major/minor breakers that ultimately create the terms that are indexed with your events. I think TERM is essentially matching your term filter against the terms found using these breakers.

This session might help (more than I will)
https://conf.splunk.com/files/2016/recordings/fields-indexed-tokens-and-you.mp4

[acharlieh]

This might be an anonymization error, or is your search that's returning 0 events on "SYSTEM 1 (AWS)" searching for the correct host?

I note that your TERM search for SYSTEM 1 is using host=indexer like the queries from SYSTEM 2, whereas the other queries in SYSTEM 1 are using host=idx (Obviously if you're trying to search for logs of a host that Splunk is neither collecting, nor distributing search to, you're going to get back no results)

The other question I have is are the two systems the exact same versions of Splunk Enterprise (we're not looking for a bug in one version or the other), and the configuration differences are not around segmentation and metrics for these logs? (although I haven't figured out exactly to make CASE work but TERM not yet, other than searching for the wrong host...)

[sansay]

Of course, it has nothing to do with the indexer name. I see that I posted one version of the query in the wrong system, because I just quickly copy pasted, not remembering that they have different indexer names.
Segmentation? I have no idea how that would impact the functionality of this command. Please elaborate.
Thank you.

Ahh and yes, both systems use the same version 7.2.4.2

[koshyk]

you may need to post the actual event message in System1 and System2 too

[sansay]

The events are pretty standard logs written by splunkd. So their content shouldn't be a determinant factor whether they are found or not by Splunk.

Anyway, here is a sample event:
"05-02-2019 23:59:56.202 +0000 INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=1.92954826907947, instantaneous_eps=6.484249982611887, average_kbps=4004.078166704291, total_k_processed=27241737783, kb=59.8125, ev=201, load_average=1.8"
Source=/opt/splunk/var/log/splunk/metrics.log
sourcetype=splunkd