Splunk Search

Can we include OR/AND operator in a transaction

amunag439
Explorer

I have the following log sets, one for success case and one for the failure case

Success:
id=11111 msg=Begin process...
id=11111 msg=check
id=11111 msg=Success...

failure:
id=22222 msg=Begin process...
id=22222 msg=check
id=22222 msg=Fail...

Here I want to check the time between the events using the transaction.

host=* sourcetype=** source="*/example.log" "Begin process*" OR "Success*"
  | transaction traceId startswith="Begin process" endswith="Success" 
  | table traceId duration _time

Above query will give me the transactions of a success case only.
Can we use AND Operator in the endswith so that I can check the duration between events irrespective of it being a success or failure?

1 Solution

niketn
Legend

@amunag439 I would recommend using stats instead of transaction for this scenario

 <yourCurrentCodeToGetTimeMsgAndID>
|  stats list(msg) as msg_all values(msg) as msg_unique min(_time) as _time max(_time) as latest_time by id
|  eval duration=latest_time-_time
|  fields - latest_time
|  eval startswith=mvindex(msg_all,0),endswith=mvindex(msg_all,mvcount(msg_all)-1)
|  fields - msg_all msg_unique
|  search startswith="Begin Process" endswith="*"

Following is a run anywhere example based on sample data and details provided. It looks only for startswith condition. Endwith any value is accepted (you can explicitly set to Success and Fail as well if there are only two final messages in your data).

|  makeresults
|  eval _time=relative_time(now(),"-1d")
|  eval data="id=11111 msg=Begin process;id=11111 msg=check;id=11111 msg=Success;id=22222 msg=Begin process;id=22222 msg=check;id=22222 msg=Fail"
|  makemv data delim=";"
|  mvexpand data
|  rename data as _raw
|  KV
|  eval msg=replace(msg,"Begin","Begin Process")
|  eval delta_duration=random()
|  eval delta_duration=substr(delta_duration,1,3)
|  accum delta_duration
|  eval _time=_time+delta_duration
|  fields - _raw delta_duration
|  stats list(msg) as msg_all values(msg) as msg_unique min(_time) as _time max(_time) as latest_time by id
|  eval duration=latest_time-_time
|  fields - latest_time
|  eval startswith=mvindex(msg_all,0),endswith=mvindex(msg_all,mvcount(msg_all)-1)
|  fields - msg_all msg_unique
|  search startswith="Begin Process" endswith="*"
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

View solution in original post

woodcock
Esteemed Legend

There is only one use case where the use of transaction is merited but this command scales so poorly that I am not even going to mention it. Stop using transaction; try this:

host=* sourcetype=** source="*/example.log" "Begin process*" OR "Success*" OR "Failure*"
| streamstats count(eval(searchmatch("Success* OR Failure*"))) AS sessionID BY traceId
| stats range(_time) AS duration list(_raw) AS events min(_time) AS _time BY traceId sessionID
0 Karma

niketn
Legend

@amunag439 I would recommend using stats instead of transaction for this scenario

 <yourCurrentCodeToGetTimeMsgAndID>
|  stats list(msg) as msg_all values(msg) as msg_unique min(_time) as _time max(_time) as latest_time by id
|  eval duration=latest_time-_time
|  fields - latest_time
|  eval startswith=mvindex(msg_all,0),endswith=mvindex(msg_all,mvcount(msg_all)-1)
|  fields - msg_all msg_unique
|  search startswith="Begin Process" endswith="*"

Following is a run anywhere example based on sample data and details provided. It looks only for startswith condition. Endwith any value is accepted (you can explicitly set to Success and Fail as well if there are only two final messages in your data).

|  makeresults
|  eval _time=relative_time(now(),"-1d")
|  eval data="id=11111 msg=Begin process;id=11111 msg=check;id=11111 msg=Success;id=22222 msg=Begin process;id=22222 msg=check;id=22222 msg=Fail"
|  makemv data delim=";"
|  mvexpand data
|  rename data as _raw
|  KV
|  eval msg=replace(msg,"Begin","Begin Process")
|  eval delta_duration=random()
|  eval delta_duration=substr(delta_duration,1,3)
|  accum delta_duration
|  eval _time=_time+delta_duration
|  fields - _raw delta_duration
|  stats list(msg) as msg_all values(msg) as msg_unique min(_time) as _time max(_time) as latest_time by id
|  eval duration=latest_time-_time
|  fields - latest_time
|  eval startswith=mvindex(msg_all,0),endswith=mvindex(msg_all,mvcount(msg_all)-1)
|  fields - msg_all msg_unique
|  search startswith="Begin Process" endswith="*"
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

amunag439
Explorer

@niketnilay in the failure case what if there are few more logs? if Fail is not a final log, how do we approach that?

0 Karma

niketn
Legend

@amunag439 the query that I have provided is similar to what transaction will do when there is starts with condition but ends with can be anything.

If you want other more specific conditions you will have to play with msg_unique which has all distinct values of msg field present for specific ids. Final 2 pipes need to change as follows.

  |  fields - msg_all
  |  search startswith="Begin Process" endswith="*"

msg_unique=Success AND msg_unique=Fail.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...