How to make multiple field extractions from my sample data?

[sunnyparmar]

Hi,

I have my logs with multiple events for which I have to make field extractions. From the first timestamp, I have to make an extraction for System.Net.WebException, from the second timestamp, I have to make for Exception:System.AggregateException, and from third timestamp, Exception: Error in DB-Logging. With this I have to show total number of counts for each exception event on an hourly basis. Can anybody guide me here on how to do this?

TIMESTAMP - [2015-09-20 21:04:25.484] THREAD ID - [44] CONTENT - Error in IsDirectoryExists/CreateDirectoryIfNotExists for directory:segro/extracts_upload/ Server: ftp://serverip Excecption: System.Net.WebException: The remote server returned an error: (550) File unavailable (e.g., file not found, no access).
TIMESTAMP - [2015-09-18 04:32:41.068] THREAD ID - [7] CONTENT - Non-recoverable Error occurred, Service will run again after 5 minutes. Exception:System.AggregateException: One or more errors occurred. ---> System.UnauthorizedAccessException: Access to the path 'File path name' is denied.
TIMESTAMP - [2015-09-16 09:15:43.508] THREAD ID - [8] CONTENT - Database Update Error for File: CurrencyExtract.txt DatabaseServer: serverip Exception: Error in DB-Logging(UpsertFtpTransfer): A network-related or instance-specific error occurred while establishing a connection to SQL Server.

Thanks

[woodcock · Accepted solution]

Like this:

... | rex "\s+Exception:\s*(?<Exception>\S+)" | timechart span=1h count by Exception

[sunnyparmar]

Hi,

Thanks for replying and the query given. it is giving result but with that it is fetching some extra fields from column like Could, NULL, Unable so could you please suggest why extraction fetching these fields? Do we need to modify more.
Thanks & Regards

[woodcock]

It is extracting those fields because you have some events that look like that. This search will show you an example of each:

 ... | rex "\s+Exception:\s*(?<Exception>\S+)" | dedup Exception | table Exception _raw

To get rid of them just tack on this:

... | fields <list of fields to keep>

[sunnyparmar]

With your first query still it is fetching that columns and yes currently i am doing in this way as you previous also told me in my some another question but i want to avoid these columns without taking help of "fields" . With this I think extraction is not giving the exact result because it is fetching the next word "could" after the "Exception" event. In my logs file in some lines after exception word there is line "could not find the file" so it is fetching "could" word in output from this line. So do you have any suggestion regarding this?
Regards

[woodcock]

If you give a sample of each log variation, we can work on updating the regex command to properly accommodate all variations.

[sunnyparmar]

Hi, any update? Thanks

[woodcock]

Switch to this:

... | rex "\s+Exception:\s*(?<Exception>[^:]+)" | dedup Exception | table Exception _raw

[sunnyparmar]

I have my logs with multiple events for which I have to make field extractions. From the first timestamp, I have to make an extraction for System.Net.WebException, from the second timestamp, I have to make for Exception:System.AggregateException, and from third timestamp, Exception: Error in DB-Logging. In fourth time stamp if you will see that is also related to DB-logging error but it has word "could" also which is fetching in your previous regex command. With this I have to show total number of counts for each exception event on an hourly basis.

 TIMESTAMP - [2015-09-20 21:04:25.484] THREAD ID - [44] CONTENT - Error in IsDirectoryExists/CreateDirectoryIfNotExists for directory:segro/extracts_upload/ Server: ftp://serverip Excecption: System.Net.WebException: The remote server returned an error: (550) File unavailable (e.g., file not found, no access).
 TIMESTAMP - [2015-09-18 04:32:41.068] THREAD ID - [7] CONTENT - Non-recoverable Error occurred, Service will run again after 5 minutes. Exception:System.AggregateException: One or more errors occurred. ---> System.UnauthorizedAccessException: Access to the path 'File path name' is denied.
 TIMESTAMP - [2015-09-16 09:15:43.508] THREAD ID - [8] CONTENT - Database Update Error for File: CurrencyExtract.txt DatabaseServer: serverip Exception: Error in DB-Logging(UpsertFtpTransfer): A network-related or instance-specific error occurred while establishing a connection to SQL Server.

TIMESTAMP - [2015-09-16 09:13:40.558] THREAD ID - [20] CONTENT - Database Update Error for File: 003701466427_SupplierExtract.txt DatabaseServer: 10.0.22.52 Exception: Error in DB-Logging(UpsertFtpTransfer): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)

Regards

[muebel]

presuming that the first event's "Excecption" is a typo somehow, you could use that pattern to setup transforms to extract that field for this sourcetype

EXTRACT-extract_ip = \sException:(?.*):

You'd want to test this out to make sure it covers the sourcetype as you'd want, but this is the general process.

[sunnyparmar]

Thanks for the reply but getting the following error if try to changed with my current extraction -

Encountered the following error while trying to update: In handler 'props-extract': Regex: unrecognized character after (? or (?-