Experience Seen: in an ES environment (though not tied to ES), a | tstats
search for an accelerated data model returns zero (or far fewer) results but | tstats allow_old_summaries=true
returns results, even for recent data.
Alternative Experience Seen: In an ES environment (though not tied to ES), running a | tstats
search in one app for an accelerated data model returns results, but there are none (or far fewer) in the other environment.
This can be caused by a mismatched datamodel search. This is not currently logged through the UI, but you can verify in search.log where you will see log messages like the following:
01-05-2016 16:42:35.818 INFO TsidxStats - Normalized search in '/opt/splunk/var/lib/splunk/defaultdb/datamodel_summary/327_813B72E7-6743-4F46-9DE6-536F78929EDD/813B72E7-6743-4F46-9DE6-536F78929EDD/DM_Splunk_SA_CIM_Malware' did not match normalized datamodel search, attempting to re-normalize
This can happen for ES environments if you have other apps installed on the same server and are running the search from one of those other apps. E.g., you have an app that defines tags (like PAN, Cisco Firewalls, etc.) but are not named with an SA-, DA-, etc. such that they are not imported by ES. The datamodel generated through ES via the CIM app will ignore those custom tags, but when you run a search from the search app, they will be present. The two searches will not match, and you will face this issue.
This can also happen for other environments where you have an app generating a datamodel where there are non-shared configurations on nested tags. If the data model includes those configs but you run the search from someplace else, that would probably surface the same behavior.
Troubleshooting and fixing this can be a bit tedious. If you have a clue to what the issue might be, it could be simpler to just experiment with disabling non-essential apps that don't match the standard ES naming scheme (SA-, DA-, TA-*, etc.).
Process:
You may need to rebuild your summaries if that change had other implications (e.g., if the datamodel was importing that app, but shouldn't have been), but may not as well.