How to ignore a section of a log in a query?

[shah_nishay]

I am parsing a file and would like to skip a section of the same
Below is the log :

| INFO | 57023 | Starting new thread FileReplayThread1, java tid: 38 system tid: 57023
| INFO | 57024 | Starting new thread FileReplayThread2, java tid: 38 system tid: 57024
| INFO | 57023 | timing=23
| INFO | 57024 | timing=24
| INFO | 57023 | timing=28
| INFO | 57024 | timing=28
| WARN | 57023 | Thread FileReplayThread1 exiting
| WARN | 57024 | Thread FileReplayThread2 exiting
| INFO | 57021 | timing=2301
| INFO | 57028 | Starting new thread FileReplayThread1, java tid: 38 system tid: 57023
| INFO | 57029 | Starting new thread FileReplayThread2, java tid: 38 system tid: 57024
| INFO | 57028 | timing=13
| INFO | 57028 | timing=84
| INFO | 57029 | timing=68
| INFO | 57029 | timing=26
| WARN | 57028 | Thread FileReplayThread1 exiting
| WARN | 57029 | Thread FileReplayThread2 exiting
| INFO | 57010 | timing=52
| INFO | 57011 | timing=53
| INFO | 57010 | timing=96

I am interested in processing "timing" value from this log but do not need any timing that comes between "Starting new thread FileReplayThread1" and "Thread FileReplayThread2 exiting"
For the above mentioned example , I am only interested in timing=2301,52,53 and 96

How can I add this in the query ?

I donot want to remove this all together from indexer as another report on the same server also uses these timings and will need all of these values.

[martin_mueller]

Assuming your pairs of starting/exiting are complete within the time range you can do something like this:

  base search yielding the events you posted
| streamstats count(eval(searchmatch("Starting new thread FileReplayThread1"))) as startCount
              count(eval(searchmatch("Thread FileReplayThread2 exiting"))) as endCount
| where startCount==endCount AND isnotnull(timing)
| further processing goes here

This keeps a running count of both messages and only keeps events with a timing field if the running counts are equal, ie each start has seen an end or vice versa.

[martin_mueller]

No start and exit messages won't be a problem. A start without an exit or an exit without a start might though.

[somesoni2]

Have you tried this??

[shah_nishay]

I wont have the Start new Thread and Thread exiting messages on all day's , it occurs only on few days and for those days I want to ignore the "timing" values that comes between these 2 lines. On some other day's logs there will only be "timing" messages without the "Start new Thread" and "exiting" lines and I need to use these data as well

[somesoni2]

Assuming each line is an event and the field extraction is not done (if fields are already extracted, remove the rex portion from the search and use the corresponding field names), try this

your base search | rex "\|\s*(?<LogType>.*)\s*\|\s*(?<ThreadId>.*)\s*\|\s*(?<Message>.*)" | transaction ThreadId startswith="Starting new thread" endswith=" exiting" keeporphans=t | where linecount=1

[shah_nishay]

this doesnt work, it still pulls up all "timings" events