Getting Data In

Finding and removing all but one duplicate doc

bruceclarke
Contributor

All,

I'm curious, is there an easy way to find all duplicate logs and delete all but one of them?

Thanks!

Tags (2)
0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

You could do something like this:

base search | streamstats count by _raw | where count > 1

That should select duplicates number 2, 3, and so on. Once you've confirmed that this really is what you're looking for, you can switch to a user with the can_delete role and pipe that to delete.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

You could do something like this:

base search | streamstats count by _raw | where count > 1

That should select duplicates number 2, 3, and so on. Once you've confirmed that this really is what you're looking for, you can switch to a user with the can_delete role and pipe that to delete.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...