Transaction command not working with events spanning more than 2 days

[nabeel652]

Hi,

We have data coming from database showing the status of Orchestrator tasks. Every tasks starts with "In Progress" and ends with "Completed". It can be in "Stuck" state for any period of time (more than 4 days in some cases). So a transaction is a task ID starting with "In Progress" and ending with "Completed" while may remain in "Stuck" State for any period of time.

index="orchestrator_tasks" | transaction ID startswith="In Progress" endswith="Completed" | where Status != "Complete"

Pretty straight forward and works fine but the problem starts when there are transactions that span over few days. I can see Splunk bundles them in Transactions "In Progress" -> "Stuck" And Then "Stuck" -> "Completed" if startswith and endswith conditions are removed. In presence of these conditions it keeps showing the first part i. e "In Progress" -> "Stuck" therefore showing such tasks as Stuck whereas they've been Completed.

My thought is that the problem is with maximum time span Splunk looks for completed event.

[Raghav2384 · Accepted solution]

maxspan is to -1 unless specified. In your case, i do not think maxspan is the issue

maxspan
Syntax: maxspan=[s | m | h | d]
Description: Specifies the maximum length of time in seconds, minutes, hours, or days that the events can span. The events in the transaction must span less than integer specified for maxspan. If the value is negative, maxspan is disabled and there is no limit.
Default: -1 (no limit)

Try adding maxevents=-1 and run the same search and see if it picks the multiple day spanning events. Since you are tracing one transaction spanning over multiple days, i doubt if the event limits is the cause. Is it possible to post a sample event set that's spanned over days? What if you hard code few IDs in the main search and see if transaction is able to catch it? Hope this leads you the right way.

Example: index="orchestrator_tasks" ID="abcdef" OR ID="ghijkl" OR ID="mnopqr"|transaction ID startswith="In Progress" endswith="Completed" maxevents=-1|where Status != "Complete"

Thanks,
Raghav

[nabeel652]

Cool. I already resolved it with maxevents=-1 and was about to update my question but your reply came through. Anyway, its correct answer!

[Raghav2384]

Excellent...glad it worked out...please accept this as answer so that it could help folks catch it easily 🙂

Thanks,
Raghav

[MuS]

@Raghav2384, please post it as answer so it can be accepted 😉 I converted your comment now.

[Raghav2384]

Thanks Michael , @MuS 🙂