Field Extraction using regex command

[ChhayaV]

Hi,

I have SharePoint logs.Here i have a field called message and I'm trying to extract the exceptions from the message.

Here is the sample entries of message(for simplicity i have numbered the messages and rest of the fields are not shown here)

1) Error warming up the worker process. - Starting worker process threw - Inner Exception: System.NullReferenceException: Object reference not set to an instance of an object. Server stack trace: at Microsoft.SharePoint.Utilities.SPShimOutOfProcUtility.GetSecurityPolicyFileCore() at Microsoft.SharePoint.Utilities.SPShimOutOfProcUtility_SubsetProxy.GetSecurityPolicyFileCore__Inner() at Microsoft.SharePoint.Utilities.SPShimOutOfProcUtility_SubsetProxy.s_GetSecurityPolicyFileCore() at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs) at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg, Int32 methodPtr, Bool
2) [SPUCWorkerProcessProxy] [SPUpgradeSession] [ERROR] [7/8/2013 9:45:31 AM]: Exception: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
3) System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error) at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket() at System.Data.SqlClient.TdsParserStateObject.ReadBuffer() at System.Data.SqlClient.TdsParserStateObject.ReadByte() at System.Data.SqlClient
4) Redirect to error.aspx failed. Exception: System.Web.HttpException: The remote host closed the connection. The error code is 0x800704CD. at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect) at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush() at System.Web.HttpResponse.Flush(Boolean finalFlush) at System.Web.HttpResponse.End() at Microsoft.SharePoint.Utilities.SPUtility.Redirect(String url, SPRedirectFlags flags, HttpContext context, String queryString)
5) Unhandled exception was thrown by the sandboxed code wrapper's Execute method in the partial trust app domain: An unexpected error has occurred. - Monitored process "ipc://eac75fb1-94c0-4941-a6e2-186e9d097b4f:7000" has encountered an unhandled exception while executing user code. - Inner Exception: Microsoft.SharePoint.UserCode.SPUserCodeSolutionProxiedException: Timeout while waiting for request to complete within the sandboxed appdomain. at Microsoft.SharePoint.UserCode.SPUserCodeApplicationHostAppDomainRef.Execute(Type userCodeWrapperType, SPUserCodeCachedAssemblyGroup userAssemblyGroup, Guid siteCollectionId, Byte[] binaryUserCodeToken, Byte[] proxyOperationToken, SPUserCodeExecutionContext executionContext)

Here i just want to extract the exceptions from each message.For example from the 1st message i want to extract "System.NullReferenceException",from 2nd "Timeout expired" ,from 3rd "System.Data.SqlClient.SqlException" and so on.
I tried with the following regex

host="sharepoint" | rex field=msg "(?i)\b exception\b:\s(?P<FieldIdentifier>[^\)]+)\:" | where FieldIdentifier !="" | table FieldIdentifier

If i do so only some of the exceptions were getting captured. So pls help with the regex.
Thank you

[MuS]

Hi ChhayaV

when I'm doing regex, I use http://gskinner.com/RegExr/ this works perfect to test regex and you see what you get.
So I took your provided data, pasted it into gskinners RegExr and tried some commands. After some try and error I came up with this regex:

 (?<=[(Innr)l ]Exception\: )(?<FieldIdentifier>.+?)\w+[\.\w]*[\s\w]+

This matches, at least for the provided data, what you asked for. Does that make sense?

Cheers, MuS

[MuS]

btw did you check out the Sharepoint App yet http://apps.splunk.com/app/1559 ?

[MuS]

couldn't stop to try further 🙂
here is another, more easy one try this:

(?<=[Il ]Exception: )(?\w+([.\s]\w+)*)

[MuS]

I think you got some starting point here and should be able to create the matching regex now.

[ChhayaV]

hi Mus,
The above regex u provided is working fine in "http://gskinner.com/RegExr/" for the given sample entries but when i use the same regex in splunk only first letter of the exception is matched.Also some matches are incorrect i.e., its matching other words which are not an exception(tried for all entries).

[ChhayaV]

Hi Ayan,if i use the above regex i'm able to capture "System.NullReferenceException" from the 1st message and "Microsoft.SharePoint.UserCode.SPUserCodeSolutionProxiedException" from the 5th message.Rest are not getting captured.

[dariusz_kwasny]

You use ':' character as the field separator. And you have no ':' character after 'Exception:' in events number 2 and 3 therefore they don't much the ":" at the end of your regexp.

[lcrielaa]

I think you'll have to use multiple regex to extract all the fields you want. For instance, using "(?<=Inner Exception: ).*?(?=:)" will catch the errors following "Inner Exception" up until the first semicolon symbol. This would extract System.NullReferenceException on the first line and Microsoft.SharePoint.UserCode.SPUserCodeSolutionProxiedException on the 5th line but not the 2nd, 3rd or 4th line. For these, you'd have to build yet another regex to capture those and make sure that they don't conflict with eachother.

Splunk uses PCRE (Perl Compliant Regular Expressions) and is pretty powerful. You can use things like positive/negative lookbehind/ahead to search for certain marker words (error, exception) and base your regex on that.

[Ayn]

It would help to know which ones are working and which ones are not.