search the result returned from "search 1" and append the second search and then pass the value of search 1 to overall search the appended search

[Akumar294]

Good Day Folks,

I have facing trouble in dealing with multisearches.
For e.g.

index="a" sourcetype="ab" field1=name1 AND ktg AND Job_Number=* |eval ktg_job=Job_Number|append[search index="a" sourcetype="ab"] |search ktg_job=*

Not to confuse, my criteria will only return one Job_Number, so i have used eval ktg_job to store that number.

The above search just returns the result that are returned by search 1, even though i have created ktg_job variable to store a number and then appended another search and finally when i am trying to search the combination of search1+search2 by passing the ktg_job=*, it just returns the result of search 1 instead of searching the whole event by looking for the value of ktg_job.

Please help on this, cannot splunk handle this capability?

[Akumar294]

Note - I have used * at all required places, but somehow in the question it is not able to display.

[Akumar294]

@493669

[493669]

not able to understand exact question..please use 101010 button while pasting any query so that no special characters will get missed
or use ctrl+k after selecting query

[493669]

can you please explain what you are trying to achieve and paste your query..

[493669]

i received your comment but i don't find on this page..
if you want to find completed status why don't you filter events using | where fieldname=completed
are you creating any saved search and scheduling every day?

[Akumar294]

Hi Rajesh, yes because i had modified that comment and again it went for evaluation.

Rajesh, that is quite a long flow with lots of complexities like the same job number but different flow, i can explain in details once i reach home.
But what i have been trying to achieve is basically, let's suppose if my events contain numbers like 105522 and if i run a search:

 index=a sourcetype=b AND 105522-->It will give me the desired result.

But if i run the below search:

index=a sourcetype=b |eval some_number=105522|search some_number=*

It fails. How we c an achieve the second scenario considering i am required to store some extracted value in eval field and the use this field value to search the entire index.

[493669]

to store this number in specific field you need to use regex to extract and store in field...if you could share your sample data then I can help with regex

[Akumar294]

Hi Rajesh,

We just want 1646600 to be extracted .But these are also in different events which have different flow, the way we have separated the flow is by looking into text "ktg". SO basically the suffix of regex will be after dot till ktg and the prefix could be .request

20180921 10:23:33 Received: Request.**1646600**.1646600^A:~{^A:~{/opt/Scripts/Summit//Informatica_Crapper.sh}~:A^^A:~{Y}~:A^^A:~{Y}~:A^^A:~{21}~:A^^A:~{Y}~:A^^A:~{stg-tes-agt-06 stg-svc-infa-tes stg-ip-etl-01.yyy.com /infadata/infa_shared/Scripts/DMS/DMS_all_channel_unzip_rename_coslog.sh **ktg**}~:A^^A:~{Y}~:A^^A:~{N}~:A^^A:~{0}~:A^^A:~{/opt/Scripts/Summit/}~:A^^A:~{1}~:A^^A:~{0}~:A^^A:~{RMS_01_UNZIP_RENAME}~:A^}~:A^

[493669]

try this regex:

|rex "Request\.(?<number>\d+)"

[Akumar294]

Hi Rajesh,

Here is the details view for your understanding:

I have logs something like below:

     20180921 10:23:33 Received: Request.1646600.1646600^A:~{^A:~{/opt/Scripts/Summit//Informatica_Wrapper.sh}~:A^^A:~{Y}~:A^^A:~{Y}~:A^^A:~{21}~:A^^A:~{Y}~:A^^A:~{stg-tes-agt-06 stg-svc-infa-tes stg-ip-etl-01.yyy.com /infadata/infa_shared/Scripts/RMS/RMS_all_channel_unzip_rename_poslog.sh ktg}~:A^^A:~{Y}~:A^^A:~{N}~:A^^A:~{0}~:A^^A:~{/opt/Scripts/Summit/}~:A^^A:~{1}~:A^^A:~{0}~:A^^A:~{RMS_01_UNZIP_RENAME}~:A^}~:A^

    . 20180921 10:23:00 Received: Request.1646472.1646472^A:~{^A:~{/opt/Scripts/Summit//Informatica_Wrapper.sh}~:A^^A:~{Y}~:A^^A:~{Y}~:A^^A:~{21}~:A^^A:~{Y}~:A^^A:~{stg-tes-agt-06 stg-svc-infa-tes stg-ip-etl-01.yyy.com /infadata/infa_shared/Scripts/RMS/RMS_all_channel_unzip_rename_poslog.sh xstore}~:A^^A:~{Y}~:A^^A:~{N}~:A^^A:~{0}~:A^^A:~{/opt/Scripts/Summit/}~:A^^A:~{1}~:A^^A:~{0}~:A^^A:~{RMS_01_UNZIP_RENAME}~:A^}~:A^

    . 20180921 10:23:45 TAProcess /bin/bash process completed with exitcode=1

 .20180921 10:23:40 TAFTPProcess /opt/cisco/TIDAL/Agent/stg-tes-agt-06b/ES_dev-tes-mtr-01_1/services/13/data/1647045.fob process completed with exitcode=0

 . 20180921 10:23:40 TAProcess /opt/Scripts/Summit//Informatica_Wrapper.sh process completed with exitcode=0.

 . 20180921 10:23:33 TAProcess /opt/Scripts/Summit//Informatica_Wrapper.sh process completed with exitcode=0.

 . 20180921 10:09:51 TAProcess /opt/Scripts/Summit//Informatica_Wrapper.sh process completed with exitcode=0.

Now, first of all i have extracted the job number by using search :

index="xcenter" sourcetype="agent"| rex field=_raw "\.(?\d\d\d\d\d\d\d)\."

In this case, job numbers extracted are for e.g. - 1646600,1646472,1647037,1647045,1646600,1646600.
As we can see, the first two events are having Job number(1646600,164672) present along with Job_Name as well(DSC="RMS_01_UNZIP_RENAME")

Although the job name is same for first two events but they have different flow("poslog.sh ktg","poslog.sh Xstore") in our architecture(That is why they have different job number).
The last four events are mainly meant for status of these jobs.
NOTE: In our environment, job number remains same for one day but changes the very next day, but flow and job name will be same always.

Now i would like to know the
1.completion status("COMPLETED WITH EXITCODE=0) for-
2.job - RMS_01_UNZIP_RENAME which is in flow - poslog.sh ktg(or ktg) which is of course the first event.
So i would further store the job number from first event(let's say in variable eval ktg_job)

index="xcenter" sourcetype="agent"| rex field=_raw "\.(?\d\d\d\d\d\d\d)\."|search DSC=RMS_01_UNZIP_RENAME AND ktg|dedup Job_Number|eval ktg_job=Job_Number

and then search with that job number in all the events by giving ktg_job=* AND "completed with exitcode=0"

 index="xcenter" sourcetype="agent"| rex field=_raw "\.(?\d\d\d\d\d\d\d)\."|search DSC=RMS_01_UNZIP_RENAME AND ktg|dedup Job_Number|eval ktg_job=Job_Number|append[search index="xcenter" sourcetype="agent"]|search "completed with exitcode=0"

But i am not able to achieve it, any help would be greatly helpful for me.

[493669]

Does this search will run on daily basis(as saved search) if yes then on what timerange?
If I understand correctly, you want to get completed with exitcode=0 events for each job number..
How you can say if below events belong to particular job number-

. 20180921 10:23:40 TAProcess /opt/Scripts/Summit//Informatica_Wrapper.sh process completed with exitcode=0.

[Akumar294]

Hi Rajesh,

Somehow the job number was not copied on couple of events(very strange):
PFB the last five events which has missing job_number: (If again same problem happens, please do let me know your contact email, i will send you over there)

9/21/18
10:23:49.000 AM 
. 20180921 10:23:49 TAProcess /opt/Scripts/Summit//Informatica_Wrapper.sh process completed with exitcode=0.
host =  prd-p-fgkhlsx58qnw source = ES_dev-tes-mtr-01_1_20180921102351.log sourcetype = agent
9/21/18
10:23:40.000 AM 
. 20180921 10:23:40 TAProcess /opt/Scripts/Summit//Informatica_Wrapper.sh process completed with exitcode=0.
host =  prd-p-fgkhlsx58qnw source = ES_dev-tes-mtr-01_1_20180921102351.log sourcetype = agent
9/21/18
10:23:33.000 AM 
. 20180921 10:23:33 TAProcess /opt/Scripts/Summit//Informatica_Wrapper.sh process completed with exitcode=0.
host =  prd-p-fgkhlsx58qnw source = ES_dev-tes-mtr-01_1_20180921102351.log sourcetype = agent
9/21/18
10:23:14.000 AM 
. 20180921 10:23:14 TAProcess /opt/Scripts/Summit//Informatica_Wrapper.sh process completed with exitcode=0.
host =  prd-p-fgkhlsx58qnw source = ES_dev-tes-mtr-01_1_20180921102351.log sourcetype = agent
9/21/18
10:23:07.000 AM 
. 20180921 10:23:07 TAProcess /opt/Scripts/Summit//Informatica_Wrapper.sh process completed with exitcode=0.

Also, the search will run on daily basis and we are going to run it by filtering for last 15-30 minutes max.

"If I understand correctly, you want to get completed with exitcode=0 events for each job number.." [Ankit]- Just the job number 1646600 which belongs to ktg flow(flow identified in first event in original question by string"ktg" and it has its job number 1646600), we want completion status(i.e. completed with exitcode=0 for that.

[Akumar294]

Yes, it is again missing the job number, please let me know your email Rajesh, i will send you the log over there.

[Akumar294]

I was able to achieve the desired result by making use of transaction command, ...|transaction Job_Number|ktg AND completed

[Akumar294]

Just like job_number=*|eval ktg_flow_job=job_number|append[index=a sourcetype=b]|search ktg_flow_job=* AND completed

[Akumar294]

Hello Rajesh,

Please first read case 2- it gives me the desired result but case 1 fails.
my job number changes everyday but it remains unique for the same day, i am trying to automate the search so that same search is valid for second day and all other days.
Basically i am looking for the completion status of Job number based on certain parameters that are defined in my search, please let me know for further inputs.

Flow goes like : searching whole data-->extracting job number-->searching for my required data-->storing the desired job_number into my own variable using eval-->appending a new search which will append the whole event again, then further used pipe to search for my specific data.

Case 1

index="ncenter" sourcetype="agent" | rex field=_raw "\.(?\d\d\d\d\d\d\d)\."|search BSC=DMS_01_UNZIP_RENAME AND ktg|dedup Job_Number|eval ktg_job=Job_Number|
append[search index="ncenter" sourcetype="agent"]|search ktg_job=* AND completed

Case 2

index="ncenter" sourcetype="agent" | rex field=_raw "\.(?\d\d\d\d\d\d\d)\."|search BSC=DMS_01_UNZIP_RENAME AND ktg|dedup Job_Number|
append[search index="ncenter" sourcetype="agent"]|search manually_putting_my_job_number AND completed

[Akumar294]

For e.g. Job_Number=* and then at search ktg_job=*