Correlate STUCK and UNSTUCK weblogic threads

[dbcase]

Hi,

I have the below events. What I need to do is correlate the execute thread (the 2nd one) with a STUCK message. That part is easy enough, where I get stumped is now correlating further with a unstuck message. What I'm hoping to have at the end is a table that shows the URL (in this example it is GET /rest/icontrol/sites/72178/rules HTTP/1.1 then a column with the word STUCK then another column with the word UNSTUCK or blank.

My effort so far is:

index=cox STUCK|rex "GET\s(?<URL>\S+)"|rex "\[STUCK] ExecuteThread:\s'(?<threadID>\S+)[']"|dedup threadID host|stats count by URL host threadID|sort host threadID

This just extracts out the URL and threadID from Stuck threads and does a simple table. I'm stuck on the matching up the correlating UNSTUCK message

Event data:

1/23/17
11:02:50.000 PM 
####<Jan 23, 2017 11:02:50 PM EST> <Info> <WebLogicServer> <ccivirpxa0721> <managedServer12> <[ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1485230570155> <BEA-000339> <[ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)' has become "unstuck".> 
host =  portal2 index = linecount = 1 source =  /var/nfs/SAT_SplunkLogs/weblogic/portal2/Portal2_managedServer12.log00477.zip:./managedServer12.log00477 sourcetype =   wls_managedserver splunk_server =   idx6.icontrol.splunkcloud.com
1/23/17
11:02:40.000 PM 
####<Jan 23, 2017 11:02:40 PM EST> <Info> <WebLogicServer> <ccivirpxa0721> <managedServer12> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1485230560319> <BEA-000339> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)' has become "unstuck".> 
host =  portal2 index = linecount = 1 source =  /var/nfs/SAT_SplunkLogs/weblogic/portal2/Portal2_managedServer12.log00477.zip:./managedServer12.log00477 sourcetype =   wls_managedserver splunk_server =   idx6.icontrol.splunkcloud.com
1/23/17
11:01:58.000 PM 
####<Jan 23, 2017 11:01:58 PM EST> <Error> <WebLogicServer> <ccivirpxa0721> <managedServer12> <[ACTIVE] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1485230518746> <BEA-000337> <[STUCK] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "652" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 652073 ms
[
GET /rest/icontrol/sites/72178/rules HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.0; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
X-format: json
X-ClientInfo: 7.3.8.77
Referer: https://portal.company.com/sp/
Cookie: JSESSIONID=F9XOmTyCrkis7WDdsku5tZO09U0_te9pCpjfHxhsAIqM30KIB53j!-695380872
Via: 1.1 10.210.192.38
X-Forwarded-For: 10.210.192.5
X-Forwarded-Host: portal.company.com
X-Forwarded-Server: 10.210.192.38
Connection: Keep-Alive
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
<snip>

[lguinn2 · Accepted solution]

Try this

index=cox "stuck" OR "unstuck"
| rex "GET\s(?<URL>\S+)"
| rex "\[(?<threadStatus>[STUCK|ACTIVE])\] ExecuteThread:\s'(?<threadID>\S+)[']"
| eval timestamp=strftime(_time."%x %X")
| eval threadStatus=if(threadStatus=="ACTIVE","Unstuck",threadStatus)
| sort _time
| stats list(timestamp) as Time list(threadStatus) as "Thread Status" by host threadID

It may not be the format that you asked for, but I think it will work!

[dbcase]

HI Iguinn,

Hmmm I think you have made it MUCH closer but the rex for threadStatus isn't working

[dbcase]

Found it this "\[(?<threadStatus>[STUCK|ACTIVE])\] needed to be modified to this "\[(?<threadStatus>[STUCK|ACTIVE]+)\]

Now sorting thru the results, will let you know if there are any other modifications....

THANKS!!!

[lguinn2]

good catch on the regular expression!

[dbcase]

Hi Iguinn,

Based upon your query (thank you!) I modified it to get what I was hoping for. The end result looks like this

index=cox stuck OR unstuck  | rex "GET\s(?<URL>\S+)"  | rex "(?<threadStatus>(STUCK|unstuck))"| rex "(?:.*?ExecuteThread:\s'){2}(?<threadID>\S+)[']"  | eval timestamp=strftime(_time,"%x %X")| sort _time| dedup threadID host _time| stats list(timestamp) as Time list(threadStatus) as "Thread Status" by host threadID|sort host threadID