Splunk Search

Join the best option?

tb5821
Communicator

I have a search that finds failed jobs from my logs. Each of those failed jobs has a job number. I'd like to then take those job numbers and get all the log lines that contain one of those job numbers. Whats the best way to do this?

Tags (2)
1 Solution

wpreston
Motivator

Assuming that something like this is your search to find the failed jobs:

index=myindex "Failed Job"

You can use it as a subsearch to provide a key (e.g. the Job Number) for another search. Then you can use transaction to group the events together by Job Number, if desired. So assuming the above, and assuming that your Job Number field in Splunk is something like jobNumber, try a search like the following:

index=myindex [search index=myindex "Failed Job" | fields jobNumber | dedup jobNumber] | transaction jobNumber

The sub search provides a list of values to your main search that are the equivalent of:

(jobNumber=000001 OR jobNumber=000002 OR jobNumber=000003 etc...)

Hope this helps!

Edit: Can you post your search commands? I'm wondering if the outer search is somehow excluding the results. Also, I can't add comments from my workplace for some reason, I can only edit my response, all my suggestions will be added here as edits.

View solution in original post

wpreston
Motivator

Assuming that something like this is your search to find the failed jobs:

index=myindex "Failed Job"

You can use it as a subsearch to provide a key (e.g. the Job Number) for another search. Then you can use transaction to group the events together by Job Number, if desired. So assuming the above, and assuming that your Job Number field in Splunk is something like jobNumber, try a search like the following:

index=myindex [search index=myindex "Failed Job" | fields jobNumber | dedup jobNumber] | transaction jobNumber

The sub search provides a list of values to your main search that are the equivalent of:

(jobNumber=000001 OR jobNumber=000002 OR jobNumber=000003 etc...)

Hope this helps!

Edit: Can you post your search commands? I'm wondering if the outer search is somehow excluding the results. Also, I can't add comments from my workplace for some reason, I can only edit my response, all my suggestions will be added here as edits.

tb5821
Communicator

I got it working here's the modified search:

index=myindex | rex "(?i)^[^[][(?P[^]]+)" | sourcetype=jobLogs [search index=myindex sourcetype=jobLogs "Job Failed" | rex "(?i)^[^[][(?P[^]]+)" | fields jobNumber | dedup jobNumber] | transaction jobNumber

0 Karma

tb5821
Communicator

Here's my current search

index=myindex sourcetype=jobLogs [search index=myindex sourcetype=jobLogs "Job Failed" | rex "(?i)^[^[]*[(?P[^]]+)" | fields jobNumber | dedup jobNumber] | transaction jobNumber

0 Karma

tb5821
Communicator

I think I found the problem somewhere in the fact that I'm using rex to obtain the jobNubmer and its not using jobNumber=###### once the subsearch is evaluated.

0 Karma

tb5821
Communicator

The sub-search by itself produces the jobNumber's as expected but for some reason adding the outer search produces zero results.

0 Karma

linu1988
Champion

You will figure it out.Please use this for reference:

http://docs.splunk.com/Documentation/Storm/Storm/User/Useasubsearch

0 Karma

tb5821
Communicator

That doesn't work I don't think you can have jobid = and then a bracket.

0 Karma

linu1988
Champion

index=myindex source=jobs jobid=[|search index=myindex source=jobs "Failed Job"|fields jobid]

0 Karma

tb5821
Communicator

It won't let me do jobid=[

Error in 'search' command: Unable to parse the search: Comparator '=' has an invalid term on the right hand side.

0 Karma

linu1988
Champion

Yes that means you will get the job id in your events, put the search in the subsearch index=myindex source=jobs jobid=[index=myindex source=jobs "Failed Job"|fields jobid]

0 Karma

tb5821
Communicator

I search: index=myindex source=jobs "Failed Job"

one of the fields that gets extracted is jobId but this obviously only gets me that one line with Failed Job not all the lines for that job.

0 Karma

linu1988
Champion

According to your post "I have a search that finds failed jobs from my logs." what does that mean? how do you know that it's a failed job event, could you tell us?

0 Karma

tb5821
Communicator

I don't want to manually have to type the failed jobid I'd like to see it be dynamic off of the jobid's that were returned by looking for ones that have failed.

0 Karma

linu1988
Champion

As you mentioned in the mail post your search gives you the failed job list.

Just write your search which gives you the all the events

your search jobid=[|search failed jobid]. this will only give you the failed job events.

tb5821
Communicator

I only want the failed job events how do I achieve that with what linu1988 suggested?

0 Karma

cpeteman
Contributor

Yeah I am have problems getting the subsearch to only give the job numbers that failed. linu1988 has the best idea I can think of.

0 Karma

cpeteman
Contributor

Append a subsearch to the search then sort by job number. I'll try and make an example.

0 Karma

tb5821
Communicator

Need more details then that...

0 Karma

cpeteman
Contributor

Actually I would say that append might be better.

0 Karma

linu1988
Champion

First extract the job number fields and then use a subsearch, filter out the failed job events.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...