Splunk Search

Is it possible to use dedup to grab the oldest events rather than the newest?

jambajuice
Communicator

I would like to dedup a series of events and save the oldest event for each host. Is it possible to use dedup for that? From what I gather, dedup will use the _time field to decide which events to keep, keeping the most recent. Is it possible to make dedup grab older events or force it to use a different field to decide which events to keep?

Thx.

Tags (1)
0 Karma

somesoni2
Revered Legend

Another way could be using stats, which I believe will be little faster than dedup:

* | stats last(*) as *, last(_*) as _* by host

The only this here is that it will shows the results as table (which in most cases desirable).

0 Karma

AnilPujar
Path Finder

If data is huge, it will take long time, and also the default limit is 200MB, if it crosses in the evaluation then you might end up in improper results.

If you are ok, with time and memory consumption.

0 Karma

David
Splunk Employee
Splunk Employee

You should be able to use the sortby +_time field to make that happen.

* | dedup MyField sortby +_time

AnilPujar
Path Finder

For me, when I searched and verified, I got the earliest event

0 Karma

marcosgarcia
New Member

I downvoted this post because sort by is applied to the results after deduping

0 Karma

Anam
Community Manager
Community Manager

Hi macrosgarcia

Downvoting should only be reserved for suggestions/solutions that could be potentially harmful to a Splunk environment or goes completely against known best practices. Simply commenting with constructive feedback on the post you are concerned with will be more beneficial for the community to learn from.

Some of the most active members in Answers have helped set the standard of how voting etiquette should work in the Splunk community which distinguishes our culture apart from other Q&A forums. Upvote early and often to give credit where it’s due for high-quality posts, comment where you think feedback needs to be given, and only downvote if something potentially dangerous is suggested. If you’re interested in seeing how this voting etiquette was developed, check out this Splunk Answers post: https://answers.splunk.com/answers/244111/proper-etiquette-and-timing-for-voting-here-on-ans.html

0 Karma

joebensimo
Path Finder

Does this work? The docs say that sort by is applied to the results after deduping. However, if the docs are correct, making sort by part of dedup doesn't make sense.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...