I am seeing thousands of errors in my splunk logs from the App for AWS in the form:
2019-02-18 15:46:34,156 level=ERROR pid=26588 tid=MainThread logger=splunk_ta_aws.modinputs.cloudwatch.handler pos=utils.py:wrapper:71 | start_time=1550504529.1, iam_role="" region="========" account="==========", datainput="cloudwatch_===========" region="us-east-1" | message="Task was interrupted by an unhandled exception."
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/utils.py", line 69, in wrapper
return func(*args, **kwargs)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/cloudwatch/handler.py", line 121, in run
metrics = self._load_metrics(checkpoint, client)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/cloudwatch/handler.py", line 128, in _load_metrics
metrics = self._discover(client)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/cloudwatch/handler.py", line 160, in _discover
for metrics in self._discovery(client):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/cloudwatch/discovery/ec2.py", line 121, in __call__
lcn = group['LaunchConfigurationName']
KeyError: 'LaunchConfigurationName'
It seems like this might be what is causing my ec2 insights page to not display any data. Is there some configuration I missed when setting up the add on?
Did you ever get this resolved?
I'm getting the same exact error on a brand new Splunk AWS add-on installation (I have restarted it since the install).
Did any of the configs for AWS TA changed, like any of the API keys. Check if all the keys are same in AWS for your inputs in Splunk AWS add-on.The message in your first line is "Task was interrupted by an unhandled exception". This message seems to be for cloudwatch input, see if something changed.
nothing should have changed, been seeing this error since I configured the addon
Did you restart after doing the config. I was getting error from python script regarding authentication and for me restarting the Heavy forwarder where my add-on was configured resolved the issue.
yep, its been restarted several times
@cemeyer2 : Any luck resolving this?
I have had the same error. I was able to eliminate that error by updating the ec2.py script as suggested by Splunk Support Team.
Update from Splunk Support:
Please try the following workaround
for group in auto_scaling_groups:
lcn = group['LaunchConfigurationName']
typename = configurations[lcn]
metric_names = self._create_metric_names(typename)
for group in auto_scaling_groups:
typename = 'c5'
metric_names = self._create_metric_names(typename)
using your corresponding instance type.
There is a known limitation of AWS Launch template instance type detection.