How do I edit my transaction search to find the queue time for different steps in my sample data?

[Laya123]

Hi,

I am working in a market research company. We will send some online surveys to some samples. We have 3 steps to put the survey online:

1. first will test the survey in local servers
2. will preview the survey in live servers
3. then finally will put the survey live

The first process will be called 'Activate', second one 'Preview', and third one 'Live', but it's not compulsory that all three steps will be there at a time for one project. Some times they will do only Activate, only Preview, or only Live.

I want to see the projects which are Activated, Preview and Live

I tried with the transaction command to get these, but was unable to put multiple startswith or endswith parameters in one transaction command.

I will give sample logs that will help to understand my issue:

2015-06-30 15:23:56,286 UTC [14] INFO 14 Logger - [52365] - ****Activate Project=AAAA, START at 15:23:56.286** -----> As i told this is here the project is 'Activate'. This will be 'Promote Project' for both Preview and Live
2015-06-30 15:23:56,288 UTC [14] INFO 14 Logger - [52365] - AAAA:: Creating Destination DPM Objects
2015-06-30 15:23:56,704 UTC [14] DEBUG 14 Logger - [52365] - (queueProject) DpmInfoManager UserName svcDIMdpm
2015-06-30 15:23:56,705 UTC [14] DEBUG 14 Logger - [52365] - (queueProject) DpmInfoManager WebServiceURL http://KOSLOMRI033/SPSSMR/ActivateWebService/ActivateWebService.asmx
.
.
.
015-06-30 15:23:57,401 UTC [14] INFO 14 Logger - [52365] - Activate Begin Activate 'AAAA' ------> I want to take this time (there will be some other projects also in queue to activate/Preview/Live. **Activate Begin Activate this will tell project is started activate) as this will give Queue time, means how much time my project will be there in queue to activate.**
2015-06-30 15:23:57,401 UTC [14] INFO 14 Logger - [52365] - AAAA:: Creating ActivateClient
2015-06-30 15:23:57,404 UTC [14] INFO 14 Logger - [52365] - ActivateClient UseWebService to True
2015-06-30 15:23:57,405 UTC [14] INFO 14 Logger - [52365] - ActivateClient Begin Activate 'AAAA'
2015-06-30 15:23:59,797 UTC [14] INFO 14 Logger - [52365] - ActivateClient Activate has been called 'AAAA'
.
.
.
2015-06-30 15:23:59,809 UTC [14] INFO 14 Logger - [52365] - Activating Project=AAAA : TaskId=207f21a3-f030-4ec2-a5c5-625ccefa2b04, SUCCEEDED
2015-06-30 15:23:59,810 UTC [14] INFO 14 Logger - [52365] - Activating Project=AAAA, END at 15:23:59.810
2015-06-30 15:23:59,812 UTC [10] INFO 10 Logger - [52365] - Post Client URL will use default credentials
2015-06-30 15:23:59,813 UTC [10] INFO 10 Logger - [52365] - Data has been posted to http://poet-na601.grpitsrv.com/POET/KO/AutoActivateLogger.asmx/Log
2015-06-30 15:23:59,829 UTC [10] INFO 10 Logger - [52365] - AfterActivationProcess called for project AAAA
2015-06-30 15:23:59,830 UTC [10] INFO 10 Logger - [52365] - Activation EndInvoke complete for project AAAA -------> this line will tell Activation completed. But for Preview and Live endswith is different **'Preview EndInvoke complete' and 'Live EndInvoke complete'.**

From the above logs, I want my output like Project Name, type of activity means Activate/Preview/Live and Start time; Queue time and End time

ProjectId   ActivationType   StartTime          DateTimeQueued    EndTime
AAAA        Activate         6/10/15 6:41 AM    6/10/15 6:41 AM   6/10/15 6:41 AM
BBBB        Preview          6/24/15 6:10 AM    6/24/15 6:10 AM   6/24/15 6:10 AM
CCCC        Live             6/29/15 10:32 PM   6/29/15 10:33 PM  6/29/15 10:32 PM

I tried with the search below, but it's not giving any results.

index=ibm sourcetype="AAS" host=AAAA | rex field=_raw "Logger - \[(?<AutoActivateId>.*)\] - " |rex field=_raw "Source TargetServer: (?<ServerName>.*)" |transaction AutoActivateId maxevents=-1 maxspan=1h startswith=eval(match(_raw,"*Activate Project\=*, START at ") OR match(_raw,"Promote Project=\=*, START at ")) endswith="*ActivateClient Begin Activate '*'" | where AutoActivateId>0 AND AutoActivateId!=777 AND AutoActivateId!=999 AND AutoActivateId!=99999 |eval weeknumber=strftime(_time,"%U") |convert ctime(_time) as Starttime |eval endtime= _time+duration |convert ctime(endtime) as Endtime|table host AutoActivateId Project Starttime Endtime weeknumber date_wday date_year date_month date_mday date_hour date_minute

Please can you help me to do this? Hope I have given a meaningful information

Thanks in advance

[emiller42 · Accepted solution]

using the following logs:

2015-06-30 15:23:56,286 UTC [14] INFO 14 Logger - [52365] - Activate Project=AAAA, START at 15:23:56.286
2015-06-30 15:23:56,288 UTC [14] INFO 14 Logger - [52365] - AAAA:: Creating Destination DPM Objects
2015-06-30 15:23:56,704 UTC [14] DEBUG 14 Logger - [52365] - (queueProject) DpmInfoManager UserName svcDIMdpm 
2015-06-30 15:23:56,705 UTC [14] DEBUG 14 Logger - [52365] - (queueProject) DpmInfoManager WebServiceURL http://KOSLOMRI033/SPSSMR/ActivateWebService/ActivateWebService.asmx
2015-06-30 15:23:57,401 UTC [14] INFO 14 Logger - [52365] - Activate Begin Activate 'AAAA'
2015-06-30 15:23:57,401 UTC [14] INFO 14 Logger - [52365] - AAAA:: Creating ActivateClient
2015-06-30 15:23:57,404 UTC [14] INFO 14 Logger - [52365] - ActivateClient UseWebService to True
2015-06-30 15:23:57,405 UTC [14] INFO 14 Logger - [52365] - ActivateClient Begin Activate 'AAAA'
2015-06-30 15:23:59,797 UTC [14] INFO 14 Logger - [52365] - ActivateClient Activate has been called 'AAAA'
2015-06-30 15:23:59,809 UTC [14] INFO 14 Logger - [52365] - Activating Project=AAAA : TaskId=207f21a3-f030-4ec2-a5c5-625ccefa2b04, SUCCEEDED
2015-06-30 15:23:59,810 UTC [14] INFO 14 Logger - [52365] - Activating Project=AAAA, END at 15:23:59.810
2015-06-30 15:23:59,812 UTC [10] INFO 10 Logger - [52365] - Post Client URL will use default credentials
2015-06-30 15:23:59,813 UTC [10] INFO 10 Logger - [52365] - Data has been posted to http://poet-na601.grpitsrv.com/POET/KO/AutoActivateLogger.asmx/Log
2015-06-30 15:23:59,829 UTC [10] INFO 10 Logger - [52365] - AfterActivationProcess called for project AAAA
2015-06-30 15:23:59,830 UTC [10] INFO 10 Logger - [52365] - Activation EndInvoke complete for project AAAA

The following search string appears to generate what you want.

sourcetype="aas" ("Activate" OR "Activation" OR "Preview" OR "Live") ("START" OR "Begin" OR "EndInvoke") 
| eval ActivationType=case(searchmatch("Activate OR Activation"), "Activate", searchmatch("Preview"), "Preview", searchmatch("Live"), "Live") 
| rex "Begin\s\w+\s'(?<Project>[^']+)" 
| rex "project\s(?<Project>[^\s]+)" 
| eval DateTimeQueued=if(searchmatch("Begin"), _time, "") 
| stats earliest(_time) as StartTime min(DateTimeQueued) as DateTimeQueued latest(_time) as EndTime by Project ActivationType 
| fieldformat StartTime=strftime(StartTime, "%c") 
| fieldformat EndTime=strftime(EndTime, "%c") 
| fieldformat DateTimeQueued=strftime(DateTimeQueued, "%c")

Approach is to ignore transactions, and instead identify the specific events we care about, and classify them with fields. Then we can just use stats on those fields to group things up. This assumes that Project is a unique identifier and only shows up once for each step.

Recommend building that search pipe by pipe to see how it progresses.

[Laya123]

Thank you so much its working

[Laya123]

Please help me to do this