Splunk Search

Transaction command not working with events spanning more than 2 days

nabeel652
Builder

Hi,

We have data coming from database showing the status of Orchestrator tasks. Every tasks starts with "In Progress" and ends with "Completed". It can be in "Stuck" state for any period of time (more than 4 days in some cases). So a transaction is a task ID starting with "In Progress" and ending with "Completed" while may remain in "Stuck" State for any period of time.

index="orchestrator_tasks" | transaction ID startswith="In Progress" endswith="Completed" | where Status != "Complete"

Pretty straight forward and works fine but the problem starts when there are transactions that span over few days. I can see Splunk bundles them in Transactions "In Progress" -> "Stuck" And Then "Stuck" -> "Completed" if startswith and endswith conditions are removed. In presence of these conditions it keeps showing the first part i. e "In Progress" -> "Stuck" therefore showing such tasks as Stuck whereas they've been Completed.

My thought is that the problem is with maximum time span Splunk looks for completed event.

0 Karma
1 Solution

Raghav2384
Motivator

maxspan is to -1 unless specified. In your case, i do not think maxspan is the issue

maxspan
Syntax: maxspan=[s | m | h | d]
Description: Specifies the maximum length of time in seconds, minutes, hours, or days that the events can span. The events in the transaction must span less than integer specified for maxspan. If the value is negative, maxspan is disabled and there is no limit.
Default: -1 (no limit)

Try adding maxevents=-1 and run the same search and see if it picks the multiple day spanning events. Since you are tracing one transaction spanning over multiple days, i doubt if the event limits is the cause. Is it possible to post a sample event set that's spanned over days? What if you hard code few IDs in the main search and see if transaction is able to catch it? Hope this leads you the right way.

Example: index="orchestrator_tasks" ID="abcdef" OR ID="ghijkl" OR ID="mnopqr"|transaction ID startswith="In Progress" endswith="Completed" maxevents=-1|where Status != "Complete"

Thanks,
Raghav

View solution in original post

Raghav2384
Motivator

maxspan is to -1 unless specified. In your case, i do not think maxspan is the issue

maxspan
Syntax: maxspan=[s | m | h | d]
Description: Specifies the maximum length of time in seconds, minutes, hours, or days that the events can span. The events in the transaction must span less than integer specified for maxspan. If the value is negative, maxspan is disabled and there is no limit.
Default: -1 (no limit)

Try adding maxevents=-1 and run the same search and see if it picks the multiple day spanning events. Since you are tracing one transaction spanning over multiple days, i doubt if the event limits is the cause. Is it possible to post a sample event set that's spanned over days? What if you hard code few IDs in the main search and see if transaction is able to catch it? Hope this leads you the right way.

Example: index="orchestrator_tasks" ID="abcdef" OR ID="ghijkl" OR ID="mnopqr"|transaction ID startswith="In Progress" endswith="Completed" maxevents=-1|where Status != "Complete"

Thanks,
Raghav

nabeel652
Builder

Cool. I already resolved it with maxevents=-1 and was about to update my question but your reply came through. Anyway, its correct answer!

0 Karma

Raghav2384
Motivator

Excellent...glad it worked out...please accept this as answer so that it could help folks catch it easily 🙂

Thanks,
Raghav

0 Karma

MuS
SplunkTrust
SplunkTrust

@Raghav2384, please post it as answer so it can be accepted 😉 I converted your comment now.

0 Karma

Raghav2384
Motivator

Thanks Michael , @MuS 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...