How to extract multiple field values

[sunnyparmar]

Hi,

I have sample logs data given below.

Now I want to make Multiple field extractions like from the first timestamp, I have to make an extraction for System.Net.WebException, from the second timestamp I have to make for Exception:System.AggregateException, and from third timestamp Exception: Error in DB-Logging. Fourth time stamp is also related to DB related error, but it has a line error: 40 - Could not open a connection to SQL Server. My current search is fetching data for all the three exceptions, but somehow regex is not doing the exact work as "Could", "Null", and "unable" fields also coming in the output, so I want to remove these fields. Any idea how to do this?

Search:

index="abc"  CONTENT sourcetype="def" | rex "\s+Exception:\s*(?<ExceptionN\S+)" | timechart span=1h count by Exception

Logs Example:

TIMESTAMP - [2015-09-20 21:04:25.484] THREAD ID - [44] CONTENT - Error in IsDirectoryExists/CreateDirectoryIfNotExists for directory:segro/extracts_upload/ Server: ftp://serverip Excecption: System.Net.WebException: The remote server returned an error: (550) File unavailable (e.g., file not found, no access).
TIMESTAMP - [2015-09-18 04:32:41.068] THREAD ID - [7] CONTENT - Non-recoverable Error occurred, Service will run again after 5 minutes. Exception:System.AggregateException: One or more errors occurred. ---> System.UnauthorizedAccessException: Access to the path 'File path name' is denied.
TIMESTAMP - [2015-09-16 09:15:43.508] THREAD ID - [8] CONTENT - Database Update Error for File: CurrencyExtract.txt DatabaseServer: serverip Exception: Error in DB-Logging(UpsertFtpTransfer): A network-related or instance-specific error occurred while establishing a connection to SQL Server.
TIMESTAMP - [2015-09-16 09:13:40.558] THREAD ID - [20] CONTENT - Database Update Error for File: 003701466427_SupplierExtract.txt DatabaseServer: 10.0.22.52 Exception: Error in DB-Logging(UpsertFtpTransfer): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)

Thanks

[Richfez]

sunnyparmar,

Two techniques for rex that I've found handy and you may too:

I see you are using \S+ to mean "not including spaces". There's a way to do that for non-special characters too. For instance, your example rex modified to include all text up to the first opening paragraph marker "(" could be...

rex "\s+Exception:\s*(?<Exception>[^(]*)"

(escaping can get tricky on those sometimes) Then, more specifically for most of your problem, you may find it more useful to pull more than one thing out of the strings at a time, this way you can anchor your extracts by everything in the event instead of just pulling out "error " which is prone to find EVERY "error noterror" string as well. It often means you end up putting these in props/transforms to do them automatically, but it works very well. It take a bit of thinking and knowing your data though. I see:

TIMESTAMP - [2015-09-20 21:04:25.484] THREAD ID - [44] CONTENT - Error in IsDirectoryExists/CreateDirectoryIfNotExists for directory:segro/extracts_upload/ Server: ftp://serverip Excecption: System.Net.WebException: The remote server returned an error: (550) File unavailable (e.g., file not found, no access).

and when I look at the other lines too (except maybe the last one), it looks to me like you have a format

TIMESTAMP - <the timestamp> THREAD ID - [<thread>] CONTENT - <error content> Exception: <the exceptionI really want>: <other error information>

The timestamp is working, I'd expect, so we'll ignore that.

rex "THREAD ID - [(?<thread>\d+)] CONTENT - (?<error_content>.*) Exception: (?<exception>[^:]*): (?<error_extra>.*)$"

Which might - it's untested because I don't have your error message in my system to easily check - pull out 4 fields. Thread (digits), error_content which should be all content between "CONTENT" and "Exception:", exception which should go from Exception: up to the next : (but exclude the colon) then everything else as "error_extra".

You will need to write one for the last of the four messages like that, but if you do, this should make all your fields be right.

Let us know how it goes, post back with an update if you can't quite get it to work from what I've provided!