Splunk Search

Join events with similar times

mstark31
Path Finder

I have an index containing failure events for both a system as a whole ("System") and individual sections of that system. When an individual section experiences a failure, two events are logged: one for the individual section and one for the system. My goal is to join the two events together (system & section) to have access to information in fields from both events. (Essentially, tag the "system" events with data from the "section" events).

About 75% of the time, the value for _time is the same for both events, which makes it easy to join them. However, sometimes the events are a few seconds apart, which means the join on time doesn't work.

How can I associate these very close (but not exactly the same time) events together?

Tags (3)
0 Karma
1 Solution

mstark31
Path Finder

I ended up coming up with a solution that takes care of the majority of cases where I want to join on time but the times aren't identical. It's a bit of a kludge, but it works most of the time.

Turns out, most of the events that don't match perfectly are within a few milliseconds of each other. I decided to create a field called joinTime where I take the value of _time and reformat it without the milliseconds ( "%F %T"). Thus, I can perform my join using the joinTime field, which is now the same for the vast majority of events that need to be combined.

I'm still on the hunt for something better, but this will work in the meantime.

View solution in original post

0 Karma

mstark31
Path Finder

I ended up coming up with a solution that takes care of the majority of cases where I want to join on time but the times aren't identical. It's a bit of a kludge, but it works most of the time.

Turns out, most of the events that don't match perfectly are within a few milliseconds of each other. I decided to create a field called joinTime where I take the value of _time and reformat it without the milliseconds ( "%F %T"). Thus, I can perform my join using the joinTime field, which is now the same for the vast majority of events that need to be combined.

I'm still on the hunt for something better, but this will work in the meantime.

0 Karma

kmorris_splunk
Splunk Employee
Splunk Employee

Is there a common value in each event that could link them? If not, is there a way you could enrich the events so that they are associated. System mapped to section for example? If there is a common value, or you could provide a common value via a lookup file and enrichment of the event, you could use the transaction command.

Transaction has parameters around the time window for a transaction (span) as well as startswith and endswith so you can identify the events that start and end the transaction. The missing link (pun intended) would be a common value between them to identify that they are related.

This could also be a third event that somehow links the other two as well.

The transaction command will group all of the related events into one transactional event.

0 Karma

mstark31
Path Finder

Thank you for your suggestion. Unfortunately, transaction isn't appropriate for this data.
I did use a variation of finding a common value in my solution.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...