Getting Data In

Duplicate Event

msn2507
Path Finder

I am extracting logs using REST webservices and its a 3rd party application that maintains the logs. I have to poll every 5 minutes to get the logs. Splunk creates duplicate events for every poll, is there a way to avoid this ?
See the sample log event -
link text

Tags (2)

Ayn
Legend

No, you need to add that logic yourself. For file-based inputs, Splunk keeps track of where to start looking for new events in the file by storing to which position it's already read the file. You need to implement the same kind of thing in your scripted input yourself - find a unique incrementing ID of some kind, then after you've queried the REST webservice store the highest ID and then next time you're querying the webservice compare event ID's against that max ID to make sure you're only getting newer ones.

msn2507
Path Finder

I am redirecting the output of the webservice call to file and made it the source for splunk still I am seeing the duplicates. Note: I have checked the option "Follow tail"

Any help is appreciated

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...