Microsoft Cloud services Addon stops collecting logs

[travis_lelle]

I've experienced the same issue in multiple environments. We're running Splunk Enterprise 6.6.3 and the Microsoft Cloud Services addon. Logs will pull for maybe a day or two, and then we begin to see the following errors in splunk_ta_microsoft-cloudservices_management.log. Typically a reboot will fix the issue, but not all the time.

  File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 240, in get_events
    self.do_get_events(content_dict)
  File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 256, in do_get_events
    events = self.get_one_content(content_dict)
  File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 154, in get_one_content
    return self._content_request(url=content_info[c.content_uri])
  File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 124, in _content_request
    raise ome.O365GetContentError(msg + http_resp.msg)
O365GetContentError: Account d3dbea26-263d-4578-bfe4-f300326a3a11_o365 [proxy_type="http" proxy_rdns="0" proxy_enabled="0" ] GET request to https://manage.office.com/api/v1.0/cc03cb3f-e51d-4fb2-b5f4-d7106
1153612/activity/feed/audit/20171031061141455019716$20171031061141455019716$audit_sharepoint$Audit_SharePoint failed, reason: 403, {"error":{"code":"AF429","message":"Too many requests. Method=GetBlob, Pu
blisherId=00000000-0000-0000-0000-000000000000"}}

2017-11-03 14:59:27,968 +0000 log_level=INFO, pid=29666, tid=Thread-70, file=o365_helper.py, func_name=request, code_line_no=102 | [proxy_type="http" proxy_rdns="0" proxy_enabled="0" ] Sending GET request
 to https://manage.office.com/api/v1.0/cc03cb3f-e51d-4fb2-b5f4-d71061153612/activity/feed/audit/20171031061205608021143$20171031061205608021143$audit_sharepoint$Audit_SharePoint
2017-11-03 14:59:27,991 +0000 log_level=INFO, pid=29666, tid=Thread-6, file=o365_content.py, func_name=tear_down, code_line_no=338 | [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Audit.SharePoint
" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" proxy_enabled="0" ]Start to tear down, wait=False
2017-11-03 14:59:27,991 +0000 log_level=INFO, pid=29666, tid=Thread-6, file=o365_content.py, func_name=tear_down, code_line_no=341 | [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Audit.SharePoint
" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" proxy_enabled="0" ]Finish to tear down, wait=False
2017-11-03 14:59:27,991 +0000 log_level=ERROR, pid=29666, tid=Thread-6, file=o365_data_collector.py, func_name=_do_safe_index, code_line_no=176 | [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Aud
it.SharePoint" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint"]Failed to get msg from servers=hf1.company.gpsvsoc.com, metric=Audit.SharePoint, error=Traceback (most recent call
 last):
O365GetContentError: [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Audit.SharePoint" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" p
roxy_enabled="0" ]Fail to get events of content 20171031061141455019716$20171031061141455019716$audit_sharepoint$Audit_SharePoint, stop this round

[travis_lelle]

Nothing. I have a support ticket open with Splunk.