Team,
In one of my setup, I have previous version of App, previous version app is unable to collect the complete sigin-in logs that is visible in Azure Portal.
As I noticed new version of app was released, Installed the new version of App in another setup,configured the inputs(used same config key from working setup) noticed the below error logs,
Kindly help to fix the issue
2019-04-25 10:27:56,177 DEBUG pid=6441 tid=MainThread file=base_modinput.py:log_debug:286 | Next URL (@odata.nextLink): https://graph.microsoft.com/beta/auditLogs/signIns?$orderby=createdDateTime&$filter=createdDateTime+...
2019-04-25 10:27:56,178 DEBUG pid=6441 tid=MainThread file=connectionpool.py:_new_conn:809 | Starting new HTTPS connection (1): graph.microsoft.com
Hi @subbarayudu,
When I faced the same problem, I could improve the collection rate to over 99% by modifying the API calls that collect events (Sign-ins and Audit Logs) as follows.
ex) When the delay time is 5 minutes
$SPLUNK_HOME/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_audit.py
50 event_source = "tenant_id:%s" % tenant_id
51 query_date = get_start_date(helper, check_point_key)
52 query_date_end = (datetime.datetime.utcnow() - datetime.timedelta(minutes=5)).strftime('%Y-%m-%dT%H:%M:%S.%fZ')
53 access_token = azauth.get_access_token(client_id, client_secret, tenant_id)
54
55 if(access_token):
56 url = "https://graph.microsoft.com/beta/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activityDateTime+gt+%s)+and+(activityDateTime+le+%s)" % (query_date, query_date_end)
57 audit_events = azutils.get_items(helper, access_token, url)
$SPLUNK_HOME/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_signins.py
52 event_source = "tenant_id:%s" % tenant_id
53 query_date = get_start_date(helper, check_point_key)
54 query_date_end = (datetime.datetime.utcnow() - datetime.timedelta(minutes=5)).strftime('%Y-%m-%dT%H:%M:%S.%fZ')
55 access_token = azauth.get_access_token(client_id, client_secret, tenant_id)
56
57 if(access_token):
58 url = "https://graph.microsoft.com/beta/auditLogs/signIns?$orderby=createdDateTime&$filter=(createdDateTime+gt+%s)+and+(createdDateTime+le+%s)" % (query_date, query_date_end)
59 sign_ins = azutils.get_items(helper, access_token, url)
There is no need to restart the Splunk service after fixing.
The corrected API call will be executed at the next collection timing, and events from the checkpoint up to 5 minutes before the acquisition timing will be collected.
If you want to check that the modified API call is being executed, you can check it from the DEBUG log of App.
ex) Sign-ins
2019-05-31 15:09:23,282 DEBUG pid=32155 tid=MainThread file=connectionpool.py:_make_request:400 | https://graph.microsoft.com:443 "GET /beta/auditLogs/signIns?$orderby=createdDateTime&$filter=(createdDateTime+gt+2019-05-31T05:58:52.8129242Z)+and+(createdDateTime+le+2019-05-31T06:04:22.013821Z) HTTP/1.1" 200 None
Hi @subbarayudu,
The issue is most likely related to the MS API itself. As a test, if you could make a call to the API separately and download the logs locally, you can then ingest these into Splunk in a separate test_index and compare against the results you have.
This Splunk Answer thread explains the situation: https://answers.splunk.com/answers/719725/input-settings-for-microsoft-office-365-reporting.html
Here’s the "gotcha" though - Microsoft may delay message trace logs up to 24 hours. During this delay, message traces may come out of sequence. Continuing our example above, a message trace log with a time stamp of 1:29 PM may have come in delayed. If we are already requesting data from 1:30 PM to 2:30 PM, we willl miss this delayed event. The delay throttle makes sure we don’t go too fast and potentially miss events.
And it's also worth checking: https://answers.splunk.com/answers/731712/time-skew-for-when-logs-are-read.html#answer-733579
Hopefully this helps explain what you're seeing. If not, please do let us know.
Hi rkantamaneni,
We do noticed splunk is unable to receive the complete logs from Azure Portal, Logs that are available in Azure Sigin logs are not forwarded to splunk, As part of validation, we searched for a user log in Azure portal and able to view activity in azure portal, but same log is not available in splunk, As part of troubleshooting we noticed during logs are being forwarded to splunk, but specific user log is not available in splunk, in splunk we even queried with Co-relation ID as well. Kindly help to address the issue.
Thanks,
Subbu
Hi @subbarayudu ,
>> previous version app is unable to collect the complete sigin-in logs that is visible in Azure Portal.
Yes, the APIs in the newer version of the app were switched over to the MS Graph API so it matches what you see in the Azure Portal vs. before.
>> 2019-04-25 10:27:56,177 DEBUG
The messages you posted seem to be DEBUG level messages rather than ERROR messages. The DEBUG messages are from the App for whatever the App makers / developer(s) decide to print out to follow the internal actions of the App. The current messages don't seem to be representative of any kind of error, were these the right messages you meant to post?
@jconger can correct me if I missed anything.