Splunk Search

How to eliminate duplicate rows in a scheduled lookup?

joydeep741
Path Finder

I have created a search to populate a lookup periodically.

 index x sourcetype=y | outputlookup abc.csv append=true

Lookup is like

EventId, Start, End
000,1,2
111,3,5

I do not want duplicate rows for EventId. My current logic is not taking care of that.
What can I add to the search so that every time a new row gets added, Splunk should only update the existing and not add a new one if event id already exists

Tags (2)
0 Karma
1 Solution

woodcock
Esteemed Legend

Like this:

index=x sourcetype=y
| appendpipe [| inputlookup abc.csv ]
| dedup EventId
| outputlookup abc.csv

You might also include _time and add before the outputlookup:

| where _time <= relative_time(now(), "-30d")

View solution in original post

woodcock
Esteemed Legend

Like this:

index=x sourcetype=y
| appendpipe [| inputlookup abc.csv ]
| dedup EventId
| outputlookup abc.csv

You might also include _time and add before the outputlookup:

| where _time <= relative_time(now(), "-30d")

somesoni2
SplunkTrust
SplunkTrust

Give this a try

index x sourcetype=y | inputlookup abc.csv append=true | dedup EventId | outputlookup abc.csv
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...