Splunk Search

Building transactions with randomly ordered events

yoho
Contributor

I have a log file with repeating patterns looking like this. Notice there are only 3 distinct field names and pay attention to the 4th, 5th and 6th lines:


Time1 field1: value1
Time2 field2: value2
Time3 field3: value3
Time4 field2: value4
Time5 field1: value5
Time6 field3: value6
Time7 field1: value7
Time8 field2: value8
Time9 field3: value9

So I have 1 interesting information per line and I would like to group these events together into a single event (probably requires usage of transactions). The result should look like this (pay attention to the 2nd line):


Time1 => Time3 - field1: value1, field2: value2, field3: value3
Time4 => Time6 - field1: value5, field2: value4, field3: value6
Time7 => Time9 - field1: value7, field2: value8, field3: value9

Unfortunately, as you can see, the order in which these fields are coming is more or less random. I can only rely on these rules, binding the lines together:
* "field3" always closes a transaction
* "field1" and "field2" have relatively close timestamps (5 minutes at most between them)

I've tried many combinations of "transaction", "filldown" and "sort" functions, but I'm unable to get the expected result.

Could somebody help me ?

0 Karma

inode
Explorer

Hi yoho, see if this article helps you shed some light:

http://foren6.wordpress.com/2014/11/18/why-is-my-splunk-transaction-not-working/

0 Karma

yoho
Contributor

I found an ankward solution.

I must first note that "filldown" function fills in the values "down the page" which is actually in the reverse order of time (as most recent events come first).

So the trick is first to reverse the order of events with "|sort _time". Then you need to add "|filldown field1, field2", so that you can propagate values of these fields to all events in the future. In the end, lines with "field3" will contain field1, field2 and field3 values and you just have to filter on these events with "|search field3".

It doesn't entirely answers to the question as you will loose the first timestamp of the transaction (Time1, Time4 and Time7) but I didn't actually really need it.

0 Karma

yoho
Contributor

Better than "|sort _time", "|reverse" is probably faster

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

If the log is written within a certain time frame consistently, you can try:

your_search | transaction maxspan=4m endswith="field3" maxevents=3 | stats...

The maxspan indicates that 4m is the time "bucket" within which all the events of the transaction fall. endswith tells transaction what the final event should be. maxevents is, well, the max number events that can be in the transaction.

Is it possible to have the log output a unique id for each transaction? That would simplify your efforts a great deal.

http://docs.splunk.com/Documentation/Splunk/5.0.3/SearchReference/Transaction

alacercogitatus
SplunkTrust
SplunkTrust

Then in that case, you'll have to have something to tie them all together, such as a field4 that has a unique identifier to tie them together.

0 Karma

yoho
Contributor

That's interesting but doesn't exactly solve the problem.

Let's take the first transaction: although Time1 and Time2 are relatively close (4m for maxspan would be ok), Time3 can be several hours later and the "maxspan" condition isn't met for the whole transaction.

I believe "transaction" can't be used to group the first 2 lines of the transaction because there is no pre-defined order but maybe another function would be ok.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...