Splunk Search

is it possible to run remove duplicates on specific dates and not on all time?

gooza
Communicator

When using the remove duplicate event python is it possible to run it on specific date range?

0 Karma
1 Solution

zpavic
Path Finder

I create new script for you. http://pastebin.com/JusgWRMy

Usage: ./splunk cmd python rem_dupl_event_spec_date.py <index> <earliest_date> <latest_date>
earliest_date & latest_date syntax: %m/%d/%Y:%H:%M:%S

For example:

./splunk cmd python rem_dupl_event_spec_date.py test 05/20/2011:0:0:0 05/24/2011:23:59:59

View solution in original post

zpavic
Path Finder

I create new script for you. http://pastebin.com/JusgWRMy

Usage: ./splunk cmd python rem_dupl_event_spec_date.py <index> <earliest_date> <latest_date>
earliest_date & latest_date syntax: %m/%d/%Y:%H:%M:%S

For example:

./splunk cmd python rem_dupl_event_spec_date.py test 05/20/2011:0:0:0 05/24/2011:23:59:59

karthy
Explorer

Hmm... I believe this script it a bit dangerous. What happens, if you have 3 events (one unique and two, which is duplicates) within the same second? Then the script will delete one of the duplicates AND the single unique event, right?

0 Karma

kphillipson
Path Finder

Really like this. Is there a way for it to only look at a source or sourcetype within a given index?

0 Karma

gooza
Communicator

works great , Thanks!

0 Karma

gooza
Communicator

Found it..
Edited the script and added to line 56 (starting with search1) earliest= latest dates :
"search1 = 'search index=' + index + 'earliest=129461760 latest=1294704000 | ..."

I don't know Perl , anyone knows how to add these dates as parameters ?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...