All Apps and Add-ons

Input settings for Microsoft Office 365 Reporting Add-on for Splunk

becksyboy
Communicator

Hi,

we are looking to define our Continuously Monitor inputs and was wondering what settings people have done for their Production deployments. I understand it can depend on the volume of message tracking logs being generated. Do we know how much data the TA can handle in terms of throughput?

We where thinking of doing the following:
Interval – every 900 seconds
Query window size – 15 mins
Delay throttle – 15 mins

thanks

1 Solution

jconger
Splunk Employee
Splunk Employee

The purpose behind the query window size and the delay throttle settings is to hedge against email message trace logs that do not come in sequentially.  For example, when the input runs, a start and end date are given to the MSFT API.  This would be the bounds for the query window.  The delay throttle is how close to “now” the end date can be. 
 
When the input runs, the largest date seen from the query is saved to use as the start date for the next run.  For example:
 
The input runs and says "give me everything between 1:00 PM and 2:00 PM".  The API gives us the information and only has data up to 1:30 PM.  No problem – we save 1:30 PM to use next time.
The input runs again and says "give me everything between 1:30 PM and 2:30 PM".  The API gives the information and we continue this cycle.
 
Here’s the "gotcha" though - Microsoft may delay message trace logs up to 24 hours.  During this delay, message traces may come out of sequence.  Continuing our example above, a message trace log with a time stamp of 1:29 PM may have come in delayed.  If we are already requesting data from 1:30 PM to 2:30 PM, we willl miss this delayed event.  The delay throttle makes sure we don’t go too fast and potentially miss events.
 
Here's a graphic to try to visualize this:
alt text

View solution in original post

manderson_rr
Explorer

Did you get your message trace data into Splunk fine? Im in the middle of the process now and Im trying to decide what to set my inputs to. How long did it take for your message trace data to fully show up?

0 Karma

becksyboy
Communicator

manderson_rr , yes we got our data in. We initially did an index once of about 25 days worth, followed by an index continuously.

It didn't take long for our data to show up.

[ms_o365_message_trace://index_once]
delay_throttle = 1
end_date_time = 2019-03-07T16:01:01
index = xxxxx
input_mode = index_once
interval = -1
office_365_password = ********
office_365_username = xxxx
query_window_size = 60
start_date_time = 2019-02-11T00:01:01

[ms_o365_message_trace://index_continuously]
delay_throttle = 5
index = xxxx
input_mode = continuously_monitor
interval = 600
office_365_password = ********
office_365_username = xxxx
query_window_size = 10
start_date_time = 2019-03-07T16:01:05

swatts1000
Explorer

So if I am understanding this right it comes down to two choices?

1) Delay reading all messages 24 hours so none are missed (by setting 24h throttle)

Or

2) Read messages quite recently (say 1 hour ago) and accept that some messages may be missed completely

Bit of a tough choice, if we are using Splunk to react to threats in email logs.

rkantamaneni_sp
Splunk Employee
Splunk Employee

Yes agreed, we'll look into creating an enhancement request to create two processes (on that checks on shorter intervals for realtime and one that checks longer intervals for historical accuracy). It could be possible to jerry rig this yourself using the add-on code if needed (or maybe duplicated the add-on twice (give a different folder name - just an idea not tested), @jconger could correct me.

0 Karma

jnowotny
Engager

Any update on this enhancement?

0 Karma

jcleary47
Path Finder

@jconger or anyone... any updates on if this app will be updated for better processing of events? We too have been having issues with the app getting behind and eventually failing to input without deleting and re-creating the input.

geraldcontreras
Path Finder

I am also having to do this regularly.
Surely there is another or better way?

0 Karma

rkantamaneni_sp
Splunk Employee
Splunk Employee

@geraldcontreras @jcleary47 @jnowotny @swatts1000 

Splunk started a new Splunk Ideas Portal (ideas.splunk.com) to post Ideas. For this issue, I've gone ahead and created one: https://ideas.splunk.com/ideas/APPSID-I-261

Please feel free to review and add comments/feedback. As a voter, you can add up to 10 votes/points per idea.

0 Karma

jconger
Splunk Employee
Splunk Employee

The purpose behind the query window size and the delay throttle settings is to hedge against email message trace logs that do not come in sequentially.  For example, when the input runs, a start and end date are given to the MSFT API.  This would be the bounds for the query window.  The delay throttle is how close to “now” the end date can be. 
 
When the input runs, the largest date seen from the query is saved to use as the start date for the next run.  For example:
 
The input runs and says "give me everything between 1:00 PM and 2:00 PM".  The API gives us the information and only has data up to 1:30 PM.  No problem – we save 1:30 PM to use next time.
The input runs again and says "give me everything between 1:30 PM and 2:30 PM".  The API gives the information and we continue this cycle.
 
Here’s the "gotcha" though - Microsoft may delay message trace logs up to 24 hours.  During this delay, message traces may come out of sequence.  Continuing our example above, a message trace log with a time stamp of 1:29 PM may have come in delayed.  If we are already requesting data from 1:30 PM to 2:30 PM, we willl miss this delayed event.  The delay throttle makes sure we don’t go too fast and potentially miss events.
 
Here's a graphic to try to visualize this:
alt text

nightflame
Explorer

Checkout this explanation link text for what can go wrong with this TA's fixed window approach and checkpoints:

https://answers.splunk.com/answers/731712/time-skew-for-when-logs-are-read.html#answer-733579

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...