Deployment Architecture

Exchange app - splunk server "oversubscribed" - aggregation and parsing queues pegged

jeff
Contributor

After installing the Splunk App for Microsoft Exchange, my "central Splunk servers" (indexers/search heads) the Aggregation Queues and Parsing Queues pegged near full for several days. After some digging, it seems to be related to the timestamp parsing for some of the powershell scripts... I was seeing a lot of errors like:

05-22-2012 10:04:17.012 -0400 WARN  DateParserVerbose - A possible timestamp match (Fri Jun 24 07:55:59 2005) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source::Powershell|host::EXMBX2|MSExchange:2010:Folder-Usage|remoteport::6919
            19 similar messages suppressed.  First occurred at: Tue May 22 09:59:16 2012
05-22-2012 10:04:17.013 -0400 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Wed Feb  1 01:02:12 2012). Context: source::Powershell|host::EXMBX1|MSExchange:2010:Folder-Usage|remoteport::2406
            16384 similar messages suppressed.  First occurred at: Tue May 22 09:59:16 2012
05-22-2012 10:04:17.280 -0400 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Sep 20 18:23:00 2008). Context: source::Powershell|host::EXMBX1|MSExchange:2010:Folder-Usage|remoteport::2406
05-22-2012 10:04:17.288 -0400 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Sep 20 18:23:00 2008). Context: source::Powershell|host::EXMBX1|MSExchange:2010:Folder-Usage|remoteport::2406

When I looked at the source for these events, there were no time stamps at all... I also did searches for the events in Splunk and came up with very few in the "current" time period... so I created a /local/props.conf file with the following attributes, the errors stopped, and the queues went back down to near zero.

[MSExchange:2010:Folder-Usage]
DATETIME_CONFIG = CURRENT

[MSExchange:2010:Mailbox-Usage]
DATETIME_CONFIG = CURRENT
0 Karma
1 Solution

ahall_splunk
Splunk Employee
Splunk Employee

Hmmm - not sure how I missed those ones when I was doing this. There is actually a big reason why we started including a date-time stamp on the events. I'll update the scripts over the weekend and post an update next week.

View solution in original post

0 Karma

ahall_splunk
Splunk Employee
Splunk Employee

Hmmm - not sure how I missed those ones when I was doing this. There is actually a big reason why we started including a date-time stamp on the events. I'll update the scripts over the weekend and post an update next week.

0 Karma

ahall_splunk
Splunk Employee
Splunk Employee

I've just released Splunk App for Microsoft Exchange v1.1.5 that corrects this problem. The Technology Add-ons have changed to emit the timestamp.

0 Karma

ahall_splunk
Splunk Employee
Splunk Employee

You've got a mismatch between the app and the Technology Add-ons that are producing the data. The app likely got updated from v1.0 to v1.1 without the deployment of the new Technology Add-ons.

All the TA's should be the same version as the Splunk_for_Exchange app.

Specifically, the new TA's include a date/time stamp, whereas the originals used DATETIME_CONFIG=CURRENT. There was a corner case for Folder-Usage and Mailbox-Usage when the folder name or mailbox name contained something that looked like a timestamp. In that case, the event was recorded erroneously.

jeff
Contributor

sample code:

function Output-FolderData($Mailbox, $Folder) {
        # Produce the output string
        $Output = New-Object System.Collections.ArrayList
        [void]$Output.add("User=`"$($Mailbox.SamAccountName)`"")
        [void]$Output.add("Folder=`"$($Folder.FolderPath)`"")
        [void]$Output.add("Type=`"$($Folder.FolderType)`"")
        [void]$Output.add("Size=$($Folder.FolderSize.ToBytes())")
        [void]$Output.add("Items=$($Folder.ItemsInFolder)")

        [string]::join(" ", $Output)
}
0 Karma

jeff
Contributor

Unfortunately no... I just installed the Exchange app last week from a fresh download and used the TAs (for Exchange 2010) that were included on the forwarders. I verified from a fresh download of 1.1.4 that my TAs match what was included. All forwarders are Universal Forwarders sending to our central indexers. The get-mailboxstats.ps1, for instance, doesn't appear to include a timestamp.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...