Splunk Search

Grouping using extracted text

newbie2tech
Communicator

Hi Team,

I am trying to extract substring from the error log and generate the stats by grouping the extracted description.

I am running into issue while groupinp and displaying them. Output is stats table.

Sample Event: lengthofEvent ErrorMessage

196  '2017-07-11 15:24:13.271 ERROR [abcd-nio-0.0.0.0-19812-abcd-37 - ErrHdlr] session 2017-07-11-10:50:44-106125199 correlationID:d8afe767-6c43-4c95-820b-1d30c1ff519c Handled abcdClientError Exception  
195  '2017-07-11 15:24:28.116 ERROR [abcd-nio-0.0.0.0-19812-abcd-5 - ErrHdlr] session 2017-07-11-10:50:44-106125199 correlationID:b6996b2e-4c35-485d-86b8-2977dd2a4a72 Handled abcdClientError Exception  
160  '2017-07-11 09:57:12.012 ERROR [abcd-nio-0.0.0.0-19727-abcd-13 - ErrHdlr] session 2017-07-11-05:48:22-922441475 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 19:32:49.239 ERROR [abcd-nio-0.0.0.0-19727-abcd-37 - ErrHdlr] session 2017-07-11-15:26:28-118543641 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 19:32:38.656 ERROR [abcd-nio-0.0.0.0-19727-abcd-33 - ErrHdlr] session 2017-07-11-15:26:28-118543641 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 19:32:16.173 ERROR [abcd-nio-0.0.0.0-19727-abcd-27 - ErrHdlr] session 2017-07-11-15:21:10-665367118 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 19:31:56.387 ERROR [abcd-nio-0.0.0.0-19727-abcd-22 - ErrHdlr] session 2017-07-11-15:26:28-118543641 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 19:28:04.655 ERROR [abcd-nio-0.0.0.0-19727-abcd-13 - ErrHdlr] session 2017-07-11-15:06:22-638369000 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 19:22:08.155 ERROR [abcd-nio-0.0.0.0-19727-abcd-45 - ErrHdlr] session 2017-07-11-15:06:22-638369000 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 19:16:32.720 ERROR [abcd-nio-0.0.0.0-19727-abcd-33 - ErrHdlr] session 2017-07-11-15:06:22-638369000 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 18:48:10.025 ERROR [abcd-nio-0.0.0.0-19727-abcd-48 - ErrHdlr] session 2017-07-11-14:46:51-745219848 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 18:21:19.248 ERROR [abcd-nio-0.0.0.0-19727-abcd-17 - ErrHdlr] session 2017-07-11-14:20:03-518907676 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 15:57:33.421 ERROR [abcd-nio-0.0.0.0-19727-abcd-18 - ErrHdlr] session 2017-07-11-11:48:10-198631538 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 14:50:04.772 ERROR [abcd-nio-0.0.0.0-19727-abcd-43 - ErrHdlr] session 2017-07-11-10:39:26-701244053 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 14:47:32.365 ERROR [abcd-nio-0.0.0.0-19727-abcd-36 - ErrHdlr] session 2017-07-11-10:39:26-701244053 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 14:43:10.549 ERROR [abcd-nio-0.0.0.0-19727-abcd-25 - ErrHdlr] session 2017-07-11-10:41:13-876681229 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 12:36:04.789 ERROR [abcd-nio-0.0.0.0-19727-abcd-44 - ErrHdlr] session 2017-07-11-08:31:13-719916592 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 12:06:44.167 ERROR [abcd-nio-0.0.0.0-19727-abcd-10 - ErrHdlr] session 2017-07-11-08:04:54-897752180 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 12:01:54.026 ERROR [abcd-nio-0.0.0.0-19727-abcd-48 - ErrHdlr] session 2017-07-11-07:59:23-662347011 correlationID: Handled abcdClientError Exception  
160  '2017-07-11 11:42:40.684 ERROR [abcd-nio-0.0.0.0-19727-abcd-24 - ErrHdlr] session 2017-07-11-07:41:27-735507332 correlationID: Handled abcdClientError Exception  
159  '2017-07-11 09:57:04.316 ERROR [abcd-nio-0.0.0.0-19727-abcd-6 - ErrHdlr] session 2017-07-11-05:48:22-922441475 correlationID: Handled abcdClientError Exception  
159  '2017-07-11 19:22:23.596 ERROR [abcd-nio-0.0.0.0-19727-abcd-1 - ErrHdlr] session 2017-07-11-15:06:22-638369000 correlationID: Handled abcdClientError Exception  

Query Used

index=abc
|eval length=len(ErrorMessage) | where length=196 OR length=195 OR length=160 OR length=159
|eval length160=substr(ErrorMessage,133,33)
|eval length159=substr(ErrorMessage,132,33) 
|eval length196=substr(ErrorMessage,169,33)
|eval length195=substr(ErrorMessage,168,33)
|eval condStr = 
if(
length=159,length159,if(
length=160,length160,if(
length=195,length195,if(
length=196,length196,ErrorMessage))))|eval condStr1=trim(condStr)| stats count by condStr1

Required Result

condStr1 count
"Handled abcdClientError Exception" 22

What i am currently getting

condStr1 count
Handled abcdClientError Exception 2

Handled abcdClientError Exception 20

The SPL is taking care of group consecutive strings (159&160 , 195&196) but not all of the strings of same description.
can you let me know what am i missing.

Tags (1)
0 Karma
1 Solution

cpetterborg
SplunkTrust
SplunkTrust

You are doing the extraction of the field the hard way. This gave me the right count:

 index=abc | rex "correlationID:\S*\s(?P<condStr1>.*)" | stats count by condStr1

A simple rex extracts the field that you want quite easily.

View solution in original post

newbie2tech
Communicator

[this is not an answer, comment has character limitation hence posting it here]
Hi cpetterborg/agarrison,

Here is my actual dashboard query, I have various errors under same index , same sourcetype. I have gone through distinct errors in the last 1month data and worked on extracting them using the substr and trying to group them as there is no recurring pattern. I am familiar with field extraction however not good with writing regex. Also listed some of the error patterns are listed below the query, unfortunately can't list them all. Also the text to be extracted is listed after the event, look for text after [EXTRACT] word.

index=abc  eventtype=err0r 
|rex field=_raw "(?<shortmessage>.*\s)" | rex field=_raw ".{65}(?<message>.{33})"
|eval length=len(shortmessage)
|eval length500=substr(_raw,280,30)
|eval length499=substr(_raw,279,30)
|eval length498=substr(_raw,280,30)
|eval length476=substr(_raw,280,30)
|eval length464=substr(_raw,178,105)
|eval length463=substr(_raw,177,105)
|eval length440=substr(_raw,277,88)
|eval length439=substr(_raw,276,88)
|eval length418=substr(_raw,244,29)
|eval length399=substr(_raw,277,122)
|eval length398=substr(_raw,276,122)
|eval length333=substr(_raw,203,51)
|eval length332=substr(_raw,202,51)
|eval length331=substr(_raw,194,36)
|eval length330=substr(_raw,193,36)
|eval length321=substr(_raw,178,88)
|eval length300=substr(_raw,176,122)
|eval length299=substr(_raw,176,121)
|eval length279=substr(_raw,245,39)
|eval length278=substr(_raw,178,79)
|eval length277=substr(_raw,175,78)
|eval length269=substr(_raw,193,53)
|eval length268=substr(_raw,192,53)
|eval length264=substr(_raw,188,74)
|eval length263=substr(_raw,187,74)
|eval length261=substr(_raw,194,33)
|eval length260=substr(_raw,193,33)
|eval length257=substr(_raw,194,25)
|eval length256=substr(_raw,193,25)
|eval length247=substr(_raw,193,54)
|eval length246=substr(_raw,192,54)
|eval length245=substr(_raw,191,54)
|eval length240=substr(_raw,180,59)
|eval length237=substr(_raw,203,33)
|eval length236=substr(_raw,202,33)
|eval length234=substr(_raw,104,50)
|eval length233=substr(_raw,193,41)
|eval length232=substr(_raw,192,41)
|eval length231=substr(_raw,184,51)
|eval length230=substr(_raw,183,48)
|eval length226=substr(_raw,192,34)
|eval length225=substr(_raw,191,34)
|eval length224=substr(_raw,188,36)
|eval length223=substr(_raw,188,35)
|eval length221=substr(_raw,188,107)
|eval length220=substr(_raw,187,107)
|eval length210=substr(_raw,186,45)
|eval length209=substr(_raw,185,45)
|eval length208=substr(_raw,184,45)
|eval length207=substr(_raw,182,25)
|eval length206=substr(_raw,192,43)
|eval length202=substr(_raw,183,57)
|eval length194=substr(_raw,51,94)
|eval length193=substr(_raw,89,102)
|eval length190=substr(_raw,157,34)
|eval length189=substr(_raw,156,34)
|eval length188=substr(_raw,181,13)
|eval length187=substr(_raw,180,13)
|eval length175=substr(_raw,99,53)
|eval length170=substr(_raw,95,54)
|eval length169=substr(_raw,94,54)
|eval length164=substr(_raw,81,83)
|eval length162=substr(_raw,96,32)
|eval length161=substr(_raw,95,32)
|eval length157=substr(_raw,95,25)
|eval length153=substr(_raw,99,55)
|eval length152=substr(_raw,112,41)
|eval length151=substr(_raw,111,41)
|eval length148=substr(_raw,94,54)
|eval length147=substr(_raw,93,55)
|eval length146=substr(_raw,92,54)
|eval length141=substr(_raw,93,54)
|eval length140=substr(_raw,92,54)
|eval length138=substr(_raw,81,56)
|eval length137=substr(_raw,81,52)
|eval length134=substr(_raw,81,53)
|eval length133=substr(_raw,84,49)
|eval length132=substr(_raw,81,52)
|eval length131=substr(_raw,80,51)
|eval length128=substr(_raw,94,43)
|eval length127=substr(_raw,94,33)
|eval length126=substr(_raw,93,33)
|eval length124=substr(_raw,87,45)
|eval length125=substr(_raw,88,45)
|eval length123=substr(_raw,98,25)
|eval length121=substr(_raw,88,117)
|eval length122=substr(_raw,89,117)
|eval length116=substr(_raw,92,35)
|eval length117=substr(_raw,93,35)
|eval length113=substr(_raw,81,42)
|eval length112=substr(_raw,80,42)
|eval length109=substr(_raw,55,64)
|eval length108=substr(_raw,54,64)
|eval length107=substr(_raw,93,43)
|eval length103=substr(_raw,66,36)
|eval length102=substr(_raw,65,36)
|eval length89=substr(_raw,68,30)
|eval length88=substr(_raw,65,30)
|eval length62=substr(_raw,137,146)
|eval length48=substr(_raw,49,42)
|eval condStr = 
if(
length=500,length500,
if(
length=499,length499,
if(
length=498,length498,
if(
length=476,length476,
if(
length=464,length464,
if(
length=463,length463,
if(
length=440,length440,
if(
length=439,length439,
if(
length=418,length418,
if(
length=399,length399,
if(
length=398,length398,
if(
length=333,length333,
if(
length=332,length332,
if(
length=331,length331,
if(
length=330,length330,
if(
length=321,length321,
if(
length=300,length300,
if(
length=299,length299,
if(
length=279,length279,
if(
length=278,length278,
if(
length=277,length277,
if(
length=268,length268,
if(
length=268,length268,
if(
length=264,length264,
if(
length=263,length263,
if(
length=261,length261,
if(
length=260,length260,
if(
length=257,length257,
if(
length=256,length256,
if(
length=247,length247,
if(
length=246,length246,
if(
length=245,length245,
if(
length=240,length240,
if(
length=237,length237,
if(
length=236,length236,
if(
length=234,length234,
if(
length=233,length233,
if(
length=232,length232,
if(
length=231,length231,
if(
length=231,length231,
if(
length=230,length230,
if(
length=226,length226,
if(
length=225,length225,
if(
length=224,length224,
if(
length=223,length223,
if(
length=221,length221,
if(
length=220,length220,
if(
length=210,length210,
if(
length=208,length208,
if(
length=209,length209,
if(
length=207,length207,
if(
length=206,length206,
if(
length=202,length202,
if(
length=193,length193,
if(
length=194,length194,
if(
length=195,length195,
if(
length=189,length189,
if(
length=190,length190,
if(
length=187,length187,
if(
length=188,length188,
if(
length=175,length175,
if(
length=169,length169,
if(
length=170,length170,
if(
length=164,length164,
if(
length=165,length165,
if(
length=162,length162,
if(
length=161,length161,
if(
length=157,length157,
if(
length=158,length158,
if(
length=153,length153,
if(
length=151,length151,
if(
length=152,length152,
if(
length=148,length148,
if(
length=147,length147,
if(
length=146,length146,
if(
length=141,length141,
if(
length=140,length140,
if(
length=138,length138,
if(
length=137,length137,
if(
length=134,length134,
if(
length=133,length133,
if(
length=136,length136,
if(
length=132,length132,
if(
length=131,length131,
if(
length=128,length128,
if(
length=127,length127,
if(
length=126,length126,
if(
length=125,length125,
if(
length=124,length124,
if(
length=123,length123,
if(
length=121,length121,
if(
length=122,length122,
if(
length=116,length116,
if(
length=117,length117,
if(
length=112,length112,
if(
length=113,length113,
if(
length=108,length108,
if(
length=109,length109,
if(
length=107,length107,
if(
length=102,length102,
if(
length=103,length103,
if(
length=88,length88,
if(
length=89,length89,
if(
length=62,length62,
if(
length=48,length48,
if(
length <48 OR (length>48 and length<62) OR (length>62 and length<88) OR (length>89 AND length<102) OR (length>103 AND length<107) OR (length>109 and length<112) OR (length>113 and length<116) OR (length>117 and length<121) OR (length>128 AND length<131)
OR (length>134 AND length<136) OR (length>138 AND length<140) OR (length>141 AND length<146)OR (length>148 AND length<151) OR (length>153 AND length<157)OR (length>157 AND length<161) OR (length>161 AND length<164) OR (length>165 AND length<169) OR (length>170 AND length<175)
OR (length>175 AND length<187) OR (length>187 AND length<189) OR (length>190 AND length<193) OR (length>194 AND length<202) OR (length>202 AND length< 206) OR (length>210 AND length<220) OR (length=222) OR (length>226 AND length<230) OR length=235 OR (length>237 AND length<240)
OR (length>240 AND length<245) OR (length>247 AND length<256) OR (length>257 AND length<260) OR length=262 OR (length>264 AND length<268)OR (length>269 AND length<277) OR (length>279 AND length<299) OR (length>300 AND length<321) OR (length>321 AND length<330) 
OR (length>333 AND length<398) OR (length>399 AND length<418) OR (length>418 AND length<439) OR (length>440 AND length<463) OR (length>464 AND length<476) OR (length>476 AND length<498) OR length>500 OR length=100 OR length=106 OR length=48 OR length=49 OR length=53,message,
message
))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))
|eval condStr=trim(condStr)
|stats count by length condStr


***Sample Events***

2017-06-10 04:15:02.982 ERROR [http-nio-0.0.0.0-19777-exec-42 - ExceptionHandlerImpl] Access denied Missing required privilleges: [restful-service]  -->[EXTRACT] Access denied Missing required privilleges
2017-06-22 17:30:12.865 ERROR [asyncTaskExecutor-7 - RentService] Async call failed to return rent account                                           -->[EXTRACT] Async call failed to return rent account
2017-07-06 03:37:27.307 ERROR [http-nio-0.0.0.0-19717-exec-5 - inteAdapter] Invalid URI encoding; using HTTP default                                 -->[EXTRACT] Invalid URI encoding
'2017-06-23 04:20:15.181 ERROR  [http-nio-0.0.0.0-19747-exec-18 - ExceptionHandlerImpl] Handled HttpServerError                                          -->[EXTRACT] Handled HttpServerError 
'2017-06-17 17:54:35.946 ERROR [http-nio-0.0.0.0-40712-exec-1 - InsideAccountsService] Failed to update accounts, retryCount: 9, time spent updating: 107.844s  -->[EXTRACT] Failed to update accounts
2017-06-17 12:05:40.487 ERROR [http-nio-0.0.0.0-40712-exec-12 - InsideAccountsService] Call doodleeRestService.isRefreshAllInProgress(1000000, accounts) failed -->[EXTRACT] doodleeRestService.isRefreshAllInProgress
2017-06-17 18:16:38.036 ERROR [http-nio-0.0.0.0-40707-exec-2 - DefaultRestExceptionHandler] Error occurred during execution of controller method. Returning status 500 org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:19) at-->[EXTRACT]500 Internal Server Error
2017-06-17 07:02:30.195 ERROR [http-nio-0.0.0.0-20011-exec-16 - LoggingHandler] Cause : com.java.app.imp.errorhandling.HttpServerErrorExtendedException: associationID: f982269c-1107-4f5a-bce0-efca885354ae SessionID: 2017-06-17-03:01:50-499526829 500 Internal Server Error Request URI: GET https://myapp-accounts:27/appsd/v1/account?context=NOT_AVAILABLE&typeIdFlags=offline_ACCOUNT Request Body: null >>>>>>>>>>>>>>>>>>>>>>>>>> RESPONSE BODY START <<<<<<<<<<<<<<<<<<<<<<<<<< {"timestamp":1497682950183,"statusCode":500,"message":"Internal Server Error","details":null} >>>>>>>>>>>>>>>>>>>>>>>>>> RESPONSE BODY END <<<<<<<<<<<<<<<<<<<<<<<<<< -->[EXTRACT]Internal Server Error Request
'2017-07-07 14:13:00.915 ERROR [http-nio-0.0.0.0-19727-exec-31 - ExceptionHandlerImpl] SessionID 2017-07-07-10:09:07-682308382 associationID: Handled HttpClientError Exception associationID: bf7a1a3c-ad36-4b18-ba63-a4d571be9632 SessionID: 2017-07-07-10:09:07-682308382 404 Not Found Request URI: GET https://appsd-community-data-service.she.pwj.com:224/ds/appsd/community/JD1234 Request Body: null >>>>>>>>>>>>>>>>>>>>>>>>>> RESPONSE BODY START <<<<<<<<<<<<<<<<<<<<<<<<<< {""timestamp"":""2017-07-07T14:13:00.863Z"",""status"":404,""error"":""Not Found"",""message"":""No message available"",""path"":""/ds/appsd/community/JD1234""} >>>>>>>>>>>>>>>>>>>>>>>>>> RESPONSE BODY END <<<<<<<<<<<<<<<<<<<<<<<<<<-->[EXTRACT] 404 Not Found Request URI
2017-06-29 19:57:28.818 ERROR [http-nio-0.0.0.0-40772-exec-38 - [dispatcherServlet]] Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.time.format.DateTimeParseException: Text '' could not be parsed at index 0] with root cause java.time.format.DateTimeParseException: Text '' could not be parsed at index -->[EXTRACT]Request processing failed; nested exception is java.time.format.DateTimeParseException
2017-07-10 20:15:21.891 ERROR [http-nio-0.0.0.0-40707-exec-3 - PrincipalPrivillegeChecker] SessionID 2017-07-10-16:05:38-805878958 associationID:67378330-709b-4228-a991-fee5efbb9a08 Entitlement response [null] has failed with error [privilleges Error - Code [400] Description [Unexpected System Error - Request timeout]] -->[EXTRACT] Entitlement response [null] has failed with error
'2017-07-01 10:46:10.164 ERROR [http-nio-0.0.0.0-20011-exec-2 - LoggingHandler] SessionID 2017-07-01-06:45:33-827946996 associationID:cf53860f-9c89-f54520ff Cause : java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.java.app.imp.errorhandling.HttpServerErrorExtendedException: associationID: cf53860f-9c89-49ed-910b-f567a74520ff SessionID: 2017-07-01-06:45:33-827946996 500 Internal Server Error Request URI: GET https://myapp-accounts:20207/appsd/v1/accountgroup Request Body: null "-->[EXTRACT]500 Internal Server Error
0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Thanks for the additional data!

The following is a rex that finds a pretty good set of condStr, though I would guess you would want to truncate some of the values back to a less unique value for statistical purposes, which can still be done, but I'm not sure what data you would want to remove from the data, since there are only 12 examples. This rex is relatively simple for this set. Here is that version:

| rex "ERROR\s+\[[^\]]+\]+\s(.*ionID:\s*[^ ]*)?(?P<condStr>.*)"

And if I were to make an uneducated guess at what might make it more useful, this would be my follow-on rex:

| rex "ERROR\s+\[[^\]]+\]+\s(.*ionID:\s*[^ ]*)?(?P<condStr>.*)" | rex mode=sed field=condStr "s/>>>.*<<<//" | rex mode=sed field=condStr "s/: [0-9]+\.[0-9]+s/:/"

This is far from perfect, but it is certainly less complex and easier to maintain and understand that trying to figure out all the little substrings you are trying to work with. Don't get me wrong, you've done an amazing job with the search you have, but I certainly would never want to follow you and figure out the problems or try to add new error types. From your search I could only get the following from the data:

176 nteAdapter] Invalid URI encoding;   1
177 InsideAccountsService] Call doodl   1
191 - ExceptionHandlerImpl] Handled H   1
191 InsideAccountsService] Failed to    1
193 cess denied Missing required privilleges: [restful-service] -->[EXTRACT] Access denied Missing requir   1
199 ] Async call failed to return ren   1
379 rincipalPrivillegeChecker] Sessio   1
396 efaultRestExceptionHandler] Error   1
458 [dispatcherServlet]] Servlet.serv   1
539 LoggingHandler] SessionID 2017-07   1
676 LoggingHandler] Cause : com.java.   1
740 ExceptionHandlerImpl] SessionID 1

which didn't seem like good data to be doing stats on. Mine isn't that much better, but it is a limited set of data:

1   404 Not Found Request URI: GET https://appsd-community-data-service.she.pwj.com:224/ds/appsd/community/JD1234 Request Body: null -->[EXTRACT] 404 Not Found Request URI
1   500 Internal Server Error Request URI: GET https://myapp-accounts:20207/appsd/v1/accountgroup Request Body: null "-->[EXTRACT]500 Internal Server Error
1   500 Internal Server Error Request URI: GET https://myapp-accounts:27/appsd/v1/account?context=NOT_AVAILABLE&typeIdFlags=offline_ACCOUNT Request Body: null -->[EXTRACT]Internal Server Error Request
1   Entitlement response [null] has failed with error [privilleges Error - Code [400] Description [Unexpected System Error - Request timeout]] -->[EXTRACT] Entitlement response [null] has failed with error
1   Access denied Missing required privilleges: [restful-service] -->[EXTRACT] Access denied Missing required privilleges
1   Async call failed to return rent account -->[EXTRACT] Async call failed to return rent account
1   Call doodleeRestService.isRefreshAllInProgress(1000000, accounts) failed -->[EXTRACT] doodleeRestService.isRefreshAllInProgress
1   Error occurred during execution of controller method. Returning status 500 org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:19) at-->[EXTRACT]500 Internal Server Error
1   Failed to update accounts, retryCount: 9, time spent updating: -->[EXTRACT] Failed to update accounts
1   Handled HttpServerError -->[EXTRACT] Handled HttpServerError
1   Invalid URI encoding; using HTTP default -->[EXTRACT] Invalid URI encoding
1   Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.time.format.DateTimeParseException: Text '' could not be parsed at index 0] with root cause java.time.format.DateTimeParseException: Text '' could not be parsed at index -->[EXTRACT]Request processing failed; nested exception is java.time.format.DateTimeParseException

The field could be more "filtered" to get better data, but I'm not sure where you are going with the data in the end - whether there are other events similar enough in the data.

I hope this is at least a little bit helpful to you. Can you share what you want to "count" from the condStr that you are getting from your searches?

0 Karma

newbie2tech
Communicator

Hi cpetterborg,

I agree my approach isn't good and is difficult to maintain and understand but my lack of rex skills made me take that route, my query does not provide proper stats data because the events which I posted I had trimmed or changed the text to hide company specific methods and information hence the variation. I am glad you have taken out time to further look into the data and sharing the rex. Your second rex with some assumption is working great pretty much extracting all the patterns except below ones which have numbers or alpha-numerics which makes the event unique and they show up as individual events causing the stats to spread across multiple pages[rowlimit=100]. Is there any way to ignore numbers or alphanumerics inside () or [] on the end of existing 2nd query to make them unique.

I just want to count the occurrence of the Error which is pretty much all events which have that word.
end of your second rex I just did | stats count by condStr

In your rex "ERROR" will this be case sensitive? can we make case-insensitive.

2nd rex output which has numbers or alphanumerics causing it to show multiple times

privilleges response [0085c9bd-b791-44d3-b868-fba063917477] has failed. 1 privilleges error(s) to report 

Call doodleeRestService.isRefreshAllInProgress(100004444, accounts) failed 

Call doodleeRestService.startRefresh(100004555) failed with com.yodlee.soap.core.refresh.refresh.InvalidRefreshModeExceptionFaultMessage: InvalidRefreshModeExceptionFaultMessage 

Call doodleeRestService.startRefreshAll(100004993) failed 

Illegal argument [CookieAuthenticationPrincipal{cookie='AMCV_73FAC51D54C72AE50A4C98BC%40AdobeOrg=-227196251%7CMCIDTS%7C17351%7CMCMID%7C61491506829527716652284815389133322820%7CMCAAMLH-1499698553%7C7%7CMCAAMB-1499698553%7CcIBAx_aQzFEHcPoEv0GwcQ%7CMCOPTOUT-1499100953s%7CNONE%7CMCAID%7CNONE; s_mtouch=DirectTypeIn; s_gnr=1499093753262-New; s_lv=1499093753266; ADRUM=s=1499353681830&r=https%3A%2F%2FKBC-beta.abc.com%2Fwma%3F0; SESSION=T0xTOjE0OTkzNTM1OTI2NTI6MTQ5OTM2Nzk5MjY1Mjp0cnVlOk9MUw; JSESSIONID=0000hbnsrhh7RwtLuguTaoS3lIp:16aoeo112; abc_KBC_AUTH=j8s6hJ2W0BtD32mR707ZyGsN53eBl1s2j6aNfP31iW622Z1R3136ePf+6I4o+o+EnHDO/x2UNSmqOCkYeXEpGk1SpEndZ1r2PjJj7iD32aWo8UGfU+sjPivmEyGqLYfzUZUag6uMLVp4MwiBFV46RBBEQnmY+OfokjdwjxvEi8ne5YrtFre8ywfm9SJGetfzUiyCnl92rNKBXCEUJ1yr9ztn/X+IG+CEaWl7bsScUywMsHSr60YlJOP+Oc6f4a79OOU6kgWI; ADRUM_BTa=R:63|g:5eb46bbb-1235-4569-8962-dd4616e667fc|n:customer1_3f5d5ee2-41b3-4e59-ad10-de0b93053644; ADRUM_BT1=R:63|i:2180|e:1|d:1', profileid='W1234', hashedprofileid='d122432bac27ddf62ba1848d219fd9ed', authLevel='10', impersonation=true, Id='8E4', sessionId='2017-07-06-11:06:24-471210352'}, com.abc.mma.accounts.dataservice.restinterface.domain.modifyname.modifyNameModificationRequest@6dbee2e8] in modifyAccounts()
0 Karma

cpetterborg
SplunkTrust
SplunkTrust

For all the GUID-type numbers you can use the following rex/sed command:

| rex mode=sed field=condStr "s/[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}//g"

and to get rid of the JSON strings, use this:

| rex mode=sed field=condStr "s/\{.*\}//"

though it may be a bit greedy. And then for the numbers, try:

| rex mode=sed field=condStr "s/\(10000\d+.*?\)//"

These might be useful for clearing out the stuff that you say is still remaining. And other such things can easily be accounted for in the search. Try them out and see if they will do what you want.

0 Karma

newbie2tech
Communicator

Thanks A lot cpetterborg, these rex did the tricks, now I pretty much have what I need. I will use your queries instead of mine for the usecase. Thx for introducing me to the power of rex there by creating urge to learn it. Thx a ton.

0 Karma

newbie2tech
Communicator

please ignore my question on "ERROR" case sensitivity, I understand it is case sensitive and we can leave it that way. Also Thanks a lot for your suggestions they have been very helpful, appreciate it.

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

You are doing the extraction of the field the hard way. This gave me the right count:

 index=abc | rex "correlationID:\S*\s(?P<condStr1>.*)" | stats count by condStr1

A simple rex extracts the field that you want quite easily.

newbie2tech
Communicator

Hi cpetterborg,

Thank you for your inputs.

I cannot use field extraction as I have 100+ different unstructured errors and we do not have any common error identifying patterns so I had to use extrapolated version of above query which i have shared. It works fine except that the rollup does not happen for all occurrences of error description and same description is being displayed more than once as specified in my sample results. I suspect it to be due to varying length(hence I used trim), will continue to check further.

Thank you for your time, your suggestion does not work for me. Any other thoughts?

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Since the example events that you provided work with my rex, but you say that the data is too unstructured and so dissimilar that you can't use my field extraction, I'm not sure how your evals can work any better. If you have more (dissimilar) data that you can share, then perhaps a better rex would work. I can tell you that I've done some pretty crazy regular expression stuff to extract the data, and it has worked quite well.

0 Karma

newbie2tech
Communicator

Hi cpetterborg/agarrison,

Please refer my below answer for details on my actual query and some sample events.

Wanted to ask if you can give a shot to above query in my original post using sample events to see if it can be achieved,thank you in advance.

0 Karma

agarrison
Path Finder

There has to be a way to group your errors or extract valuable fields. Or maybe even split it into different buckets so you can do that.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...