Summary index search has results but no logs in the index

[tkwaller]

Hello I have a savedsearch that is populating a summary index.

index=apache_access_logs OR index=jbossweb app_pool=mci (uri_path="/solr/inventory*" OR uri_path="/solr/orders*" OR uri_path="/solr/payments*") clientip!=(IP) | eval internal=if(eventtype="site_traffic_exclude_SH_ip", "external", "internal") | stats count AS totalhits dc(clientip) AS count by internal, uri_path

index=summary
Add fields:
sh_stats=unified_mci_ip_count

Here is the job in the list, as you can see there are over 34 MILLION events but none of these are in the summary index

Dispatched at   Owner   Application Size    Events  Run time    Expires Status  Actions
3/23/16 7:43:03 PM  splunk-system-user  summary_unified     0.28MB  34,424,750  00:05:31    Mar 25, 2016 7:48:37 PM Done    Inspect | Save | Delete
MCI_IP_count [earliest time=3/22/16 7:00:00 PM, latest time=3/23/16 7:00:00 PM]

Any thoughts as to WHY there would not populate the index? Thanks for the help!

[tkwaller · Accepted solution]

So after some DEEP DIGGING I found the issue. It seems that there was a stanza created by a predecessor in a props app that sent all logs for a specific host to a different index. this stanza grabbed all of our summary index data and put it in a different index so data was not in the index I was searching in.

[woodcock]

What makes you believe the data did not go in? What do you get from this search:

index=summary

[tkwaller]

I get lots of results that are from other index summary searches that go to the same index. This one has a field added sh_stats=unified_mci_ip_count but that does not exist anywhere.

Also I can run index=summary and the search_name isn't the search above

[snoobzilla]

You are using a stats aggregation command which means only the aggregated data would go into the summary index. If you are looking for the events to also be in the summary index, they won't be there.

Try searching index=summary and look for the results of your saved search that way... you should see events at time search runs with results of | stats count AS totalhits dc(clientip) AS count by internal, uri_path... I believe there is also a field added for savedsearch name in by default during summarization.

PS... I hope I am understanding your question correctly. If not, please disregard.

[tkwaller]

"Try searching index=summary and look for the results of your saved search that way... you should see events at time search runs with results of | stats count AS totalhits dc(clientip) AS count by internal, uri_path"

Nope, there are none

[woodcock]

This will happen when the SI does not exist but it should generate warnings on the Search Head in the Messages section complaining about receiving events for an index that does not exist.

[tkwaller]

But the SI does exits and has results from other searches that populate this index.