Why is the Splunk Add-on for Microsoft Office 365 Add-on duplicating log data?

[tommoore]

I've noticed that this Add-on simply pulls the last 7 days of log data from the Azure API, but makes no effort to create a marker. Thus when I enable pulls, it just duplicates the log data over and over. I've tested this by pulling in the Audit.Exchange blobs and by watching a 2 hr time period in the past. I can see the event count going up for that range every time the Add-on fires. I also can run a streamstats to check for duplicate _raw,_time events, and I see them as well.

Please advise.

[rkantamaneni_sp]

Feel free to file a support case with Splunk Support and reference ADDON-20076 .

The current workaround:

Locate the lines from 117 - 119 in file splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py, then change line 117 like this:

now = self._now() // 86400 * 86400 + 86400
end_time = datetime.utcfromtimestamp(now)
start_time = end_time - timedelta(days=7)

The following is added on line 117: // 86400 * 86400 + 86400

This should limit the duplicates, however, Microsoft still duplicates events on their side that this doesn't catch (by design), so using this search can help eliminate seeing duplicates (but doesn't help on the ingestion side):

sourcetype="o365:management:activity" | dedup _raw 

NOTE: If you encounter issues hitting Error AF20055 (https://answers.splunk.com/answers/717202/python-script-errorscode-255-and-invalid-start-tim.html) , then try changing the number of days in the last line of the script change to 6 days, e.g. days=6.

A fix to limit/prevent duplicate ingestion is in the works, but a timeline isn't available at the moment as this involves working with evolving O365 Management APIs.

Please do reply if you can and let us know if the above change helps reduce your duplicates.