Why are we experiencing problems ingesting logs after 2.0?

[wstarowicz]

Hi, after upgrade to 2.0 version, logs from signins are not ingested (we're using only this input so far). Logs show following error:

2019-10-14 12:52:52,437 ERROR pid=5027 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_signins.py", line 84, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_signins.py", line 77, in collect_events
    sign_ins = azutils.get_items(helper, access_token, url)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
    raise e
HTTPError: 429 Client Error:  for url: https://graph.microsoft.com/beta/auditLogs/signIns?$orderby=createdDateTime&$filter=createdDateTime+...

[wstarowicz]

Hi, i didn't check this setting as it started to work during night...

[jconger]

HTTP code 429 indicates "too many requests" to the Microsoft API. Try setting the query limit parameter in the input to limit the number of requests on each run.

[andrewtrobec]

@jconger I am facing this problem in 2022.  Using Microsoft Azure Add-on for Splunk 3.2.0 and Splunk Enterprise 8.2.6.

I am using "Azure Metrics" inputs, there are 48 in total and they are scheduled to run every 300 seconds (5 minutes).  I have configured 1 thread per input, so technically I am making 48 calls every 5 minutes.

I was hoping that I could modify the "Interval" parameter so it could be a cron job, which means I could run the inputs at different scheduled times, but that doesn't seem to be an option.  I was also hoping that there would be a "retry" option so that in case of Error 429 it would wait and retry, but this is not available either. 

Are there any recommended approaches for solving this issue?

The exact error I am receiving is the following:

requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://management.azure.com/providers/Microsoft.ResourceGraph/resources?api-version=2019-04-01

 
Regards,

Andrew

[andrewtrobec]

Adding the edit to mention that I figured out that in the inputs.conf file you can use a cron schedule for the "interval" parameter.  It's the TA UI that does not accept a cron value, which I think should be changed.  I see a new TA version was released in July, maybe it already accepts cron.

Anyways, my solution was to spread the inputs across the following crons:

0-59/5 * * * *
1-59/5 * * * *
2-59/5 * * * *
3-59/5 * * * *
4-59/5 * * * *

My assignment to these crons was based on which subscriptions send back the most data.

So far I am not receiving any 429 Client Error.

Regards,

Andrew

[fed_kerr]

I've got the same issue. did you fix it by setting the query limit parameter?