Why has AWS Data stopped coming into Splunk suddenly with error "ERRORClient is not authenticated"?

[brent_weaver]

We were getting cloud trail and config until 10am yesterday. I looked at events around this time in Splunk and do not see anything. We are getting the following errors.

06-12-2017 19:59:09.083 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_config.py" ERRORClient is not authenticated

and

06-12-2017 19:59:08.920 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_cloudtrail.py" ERRORFail to load AWS Accounts - {"messages":[{"type":"WARN","text":"call not properly authenticated"}]}

Does anyone have any thoughts? Where do we even begin? I am told the cloud team made no changes, but that does not mean that they didn't.

Thanks!

[lguinn2]

1. Was this message generated from within the script aws_cloudtrail.py? If so, where and why would the script issue this message?
2. Did a password change? (if passAuth is set in inputs.conf, then did the password for that user change - or expire?)
3. Did an AWS password change?
4. Did a firewall rule change?
5. Was there an update to the OS or AWS or any other piece of software?

Go to Settings -> General Settings and change the log level for the exec processor to DEBUG (I assume that it is set to INFOR or WARN now). Let it run for a bit and then see what you can find in the splunkd.log (Note that this setting will revert if you restart Splunk...)

That's where I would start...