We ran into this same issue when upgrading to 8.1.2 (and ES to 6.4.1). It looks to be related to changes in the Health Report Manager, as there are some new feature indicators. You can disable or modify the thresholds in the UI or in health.conf (see https://docs.splunk.com/Documentation/Splunk/8.1.2/DMC/Configurefeaturemonitoring). If you look at the internal logs, you can see if delayed searches have actually increased: index=_internal component=savedsplunker reason=* | timechart count  As long as no other changes were introduced, I would expect the delayed search count to stay relatively stable. Our next step is to dig in to these searches to see if we can optimize. ... View more

If you are trying to test the splunk-cloudwatch-logs-processor lambda function, the default test event will fail with "Cannot read property 'data' of undefined". We had the same problem when trying to set this up for the first time. We discovered that the lambda function is expecting the json from CloudWatch and not the default test event. The CloudWatch json has a data field that the function decodes from base64. Replacing the test event with the json below should work as a test. { "awslogs": { "data": "QVdTIGxvZyBjb250ZW50" } }   ... View more

We had the same problem when trying to set this up for the first time. We discovered that the lambda function is expecting the json from CloudWatch and not the default test event. The CloudWatch json has a data field that the function decodes from base64. Replacing the test event with the json below should work. { "awslogs": { "data": "QVdTIGxvZyBjb250ZW50" } }   ... View more

In my test environment, where I have been playing around with Splunk_TA_windows (upgrading to 5.01 and then 6.00) and Splunk App for Windows Infrastructure 1.5.2, I found that I had an old windows_apps.csv file in the App for Windows Infrastructure lookups directory. It appeared to be from a previous install as 1.5.2 does not include it. After installing TA 5.01, it updated the lookup definition, but it was still pointing to the old windows_apps.csv file which has the old "sourcetype" field. The new lookup definition looks at "source", so it couldn't complete the lookup. The error in the search log pointed me in the right direction - "Could not find all of the specified lookup fields in the lookup table". I think it was grabbing the older lookup file, because of the configuration precedence rules, but don't quote me on that. I deleted the old windows_apps.csv file from the App for Windows Infrastructure and now the lookup is using the new windows_apps.csv located in the TA lookups directory. ... View more