Tons of network errors in Python log

[goodsellt]

I'm receiving a ton of errors in my Python log from the AWS Addon on the HF I'm running the data gathering from. I have been collecting data, and seemingly my dashboards appear to be working OK. However, I'm concerned that this error is indicative of a problem which may be causing data to go missing.

Have any of you encountered this error and have successfully troubleshooted it? Is there an option in the app or on Splunk I can turn on to increase the verboseness of the logging? Since I'm unable to see which of the data gathering commands is spawning this error I'm not really sure what to do. Thanks!

Supporting info:
- Splunk 6.3.0.1
- Splunk Addon for AWS 4.1.1
- No proxy configured

Error:

2016-11-03 13:42:28,920 ERROR Caught exception reading instance data
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/boto/utils.py", line 210, in retry_url
    r = opener.open(req, timeout=timeout)
  File "/opt/splunk/lib/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/opt/splunk/lib/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/opt/splunk/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/opt/splunk/lib/python2.7/urllib2.py", line 1227, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/opt/splunk/lib/python2.7/urllib2.py", line 1197, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 101] Network is unreachable>

[sloftis2016]

From what I can tell - The retry_url function in ./boto/utils.py file is used for accessing the metadata service on an EC2 instance. Basically if you have access to an EC2 instance.. and AWS SDK for python is installed.. you can run this function and point it at the url of 169.254.169.254. Since you probably aren't running the AWS app from an EC2 instance you are going to get the URLError.

How i fixed it.
from dir /opt/splunk/etc/apps/splunk_app_aws/bin/boto
[splunk]$ cp utils.py utils.py.bak
[splunk]$ vi utils.py
add the following 2 lines to catch the exception
except urllib.error.URLError as e:
return ''

This will go in the retry_url() function between the other 2 except statements..

except urllib.error.HTTPError as e:
code = e.getcode()
if code == 404 and not retry_on_404:
return ''

except urllib.error.URLError as e:
return ''

except Exception as e:
boto.log.exception('Caught exception reading instance data')

Hope this helps - I stopped getting the error messages. This isn't the best fix but I'm new to python and splunk.
No restart or anything required.

[goodsellt]

Thanks for pointing all of this out! That's some good digging into the problem, and it lines up with some of the things I'm seeing. I'll see if I can figure out what's going on further, since most of the TA scripts call that retry function in the event of a failure, either its a defect with the SDK or potentially with the usage of the boto library by the TA itself.

[bmacias84]

Hello goodsellt,

The error is a failure to connect to AWS, which is most likely tied network connectivity issues such as an unreliable connection. Also AWS does have connection problems periodically. You are probably not losing data unless what ever data was being downloaded was unfinished.

[goodsellt]

Thanks! Would you know of any way I can check the exact function being called when this error is thrown? In an interest of thoroughness I'm attempting to find out if these errors are being generated by a single one of the data inputs or by a variety of them.