I am running a search for multiple events over a range of time. In that search, I want to only find events of one specific type if they occur 1 minute after the start of the time range.
The problem I am trying to solve here is dealing with time drift when working with transactions. For example, I have the events A, B, C which compose a transaction, but time drift will occasionally cause C to show up before A in Splunk: C, A, B. This causes my search to not find the final event in the transaction. My thought is that, if I only find transactions where the first event, A, happens 1 minute after the beginning of the search range then that will provide a buffer to deal with the time drift.
A sample set of event (This shows the clock skew, ServiceA at the bottom is actually where the chain of events begins, but the clockskew on our VMs causes it to show up last):
2015-12-09 19:12:25.5499|INFO|ServiceB|action=Received|corId=inventory_ceab9c8b6d694a09a9a10932097e632e|msgId=c87c2760468d45c0acbc246f25393ef3|parentId=8a3a1c5734414309ba6697a3ade10956
2015-12-09 19:12:25.5813|INFO|ServiceC|action=Sent|corId=inventory_ceab9c8b6d694a09a9a10932097e632e|msgId=c87c2760468d45c0acbc246f25393ef3|parentId=8a3a1c5734414309ba6697a3ade10956
2015-12-09 19:12:28.1707|INFO|ServiceA|action=Sent|eventType=inventoryUpdate|corId=inventory_ceab9c8b6d694a09a9a10932097e632e
... View more