Why does subsearch throw an exception?

[mperrenoud]

I'm trying to produce a subsearch based off of a rex field. The goal of this search is to find every Deserialization Successful log where the MessageSequence is in the subsearch list of errors. The ultimate goal there is to identify if it's the same record failing over and over.

However, the following search results in this error:

Error in 'fields' command: Invalid argument: 'seq=23080948093779922'

index="b2brtr_prod_main" sourcetype=applog "Deserialization Successful" |
    rex "MessageSequence:\s+(?<seq>\d+)" | fields _raw, seq
[search index="b2brtr_prod_main" sourcetype=applog "B2B_EXCEPTION_PROCESSMESSAGE" |
    rex "MessageSequence:\s+(?<seq>\d+)" | fields + seq]

An example B2B_EXCEPTION_PROCESMESSAGE log might be something like this:

2017-09-14 07:23:20,453-04:00 thread=12 logLevel=ERROR component=... message=... : B2B_EXCEPTION_PROCESSMESSAGE - MessageSequence: 68116944367487730 MessageException: System.AggregateException: One or more errors occurred. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) ...
---> (Inner Exception #0) System.Threading.Tasks.TaskCanceledException: A task was canceled.<---

[lfedak_splunk]

Hey @mperrenoud, If DalJeanis solved your problem, please don't forget to accept an answer! You can upvote posts as well. (Karma points will be awarded for either action.) Happy Splunking!

[DalJeanis · Accepted solution]

Updated - No, that subsearch, if it works, is going to return a lot of nonsense. You don't have anything specifically joining it to the prior part. Once we saw that, it made more sense to refactor the whole thing.

Try this way...

 index="b2brtr_prod_main" sourcetype=applog ("Deserialization Successful" OR "B2B_EXCEPTION_PROCESSMESSAGE")
 | fields _raw
 | rex "MessageSequence:\s+(?<seq>\d+)" 
 | rename COMMENT as "the above gets all the messages of both types and extracts the seq."

 | rename COMMENT as "Now we extract the message and shorten it to 3 digits (DES or B2B)"
 | rex "(?i)(?<rectype>Deserialization Successful|B2B_EXCEPTION_PROCESSMESSAGE)"
 | eval rectype=upper(substr(rectype,1,3))

 | rename COMMENT as "count up how many of each, use xyseries to assign count to 3-digit message name"
 | stats count by seq rectype
 | xyseries seq rectype count

 | rename COMMENT as "Get rid of any records that don't have both."
 | where B2B > 0 and DES > 0

You can change the tests in the last line to whatever number you want.

---

Hmmm. It's complaining about one particular seq value, and it is numeric, and not surrounded by quotes.

try this...

index="b2brtr_prod_main" sourcetype=applog "B2B_EXCEPTION_PROCESSMESSAGE" 
| rex "MessageSequence:\s+(?<seq>\d+)" 
| eval seq=tostring(seq)
| where like(seq,"230809%")
| table seq
| format

Make sure the output seq= is surrounded by quotes.

If so, then do this...

Updated to work, but use the above anyway -

 index="b2brtr_prod_main" sourcetype=applog "Deserialization Successful" 
| rex "MessageSequence:\s+(?<seq>\d+)" 
| fields _raw, seq
| eval seq=tostring(seq) 
| join seq 
    [ search index="b2brtr_prod_main" sourcetype=applog "B2B_EXCEPTION_PROCESSMESSAGE" 
    | rex "MessageSequence:\s+(?<seq>\d+)" 
    | eval seq=tostring(seq) 
    | table seq ]

[mperrenoud]

I can see I've quite a bit of work to get to this point with my Splunk queries. Thanks so much for your help!

[DalJeanis]

@mperrenoud - there are other ways to do it, that might be easier to read at your level. None of these are particularly more efficient than each other, so take your pick. The rule here is, use whatever method you will understand when you come back to modify it later.

  | rename COMMENT as "count up how many of each, assign count to 3-digit message name"
  | stats count by seq rectype
  | eval B2B=if(rectype="B2B",count,0)
  | eval DES=if(rectype="DES",count,0)

  | rename COMMENT as "stats B2B and DES values onto a seq record, then get rid of any records that dont have both."
  | stats sum(B2B) as B2B sum(DES) as DES by seq
  | where B2B > 0 and DES > 0

---

  | rename COMMENT as "count up how many of each, assign count to 3-digit message name"
  | stats count by seq rectype
  | eval {rectype} = count

  | rename COMMENT as "get rid of rectype field, stats B2B and DES values onto a seq record, and then get rid of any records that dont have both."
  | fields - rectype
  | stats values(*) as * by seq
  | where B2B > 0 and DES > 0