Splunk Search

How to remove all identical events and keep the latest event for each different timestamp in a transaction search?

kkarthik2
New Member

We tried this search below:

index=test | eval dup=_raw | convert ctime(_time) as T1 |  transaction dup mvlist=t  maxspan=1s keepevicted=true | where eventcount>1

And these are sample events:

_time : 4/9/2015 12:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
_time : 4/9/2015 12:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
_time : 4/9/2015 12:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
time : 4/9/2015 13:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
time : 4/9/2015 13:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
time : 4/9/2015 14:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
time : 4/9/2015 15:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.

I want latest events like the example below, remove the other events without using dedup command,
time : 4/9/2015 15:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Another strategy for this one, faster than dedup if the same fields are consistently present:

 index=test 
| stats  latest(_time) as _time latest(_raw) as _raw by host source sourcetype Date status Name 

...or ...

 index=test 
| stats  latest(_time) as _time by _raw

...but the second one means you need to re-extract the other fields.

0 Karma

devin_stonecyph
Explorer

Why do you want to avoid using dedup? Dedup is by far the easiest way of doing this. Just |dedup host, source, sourcetype, Date, Name | , and that's all you need to do. Add that before the transaction.

Dedup will leave you with the most recent result from each duplicate set.

From the docs: "Events returned by dedup will be the first event found in search order, which is the most recent for historical search"

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...