Getting Data In

Duplicate Event

msn2507
Path Finder

I am extracting logs using REST webservices and its a 3rd party application that maintains the logs. I have to poll every 5 minutes to get the logs. Splunk creates duplicate events for every poll, is there a way to avoid this ?
See the sample log event -
link text

Tags (2)

Ayn
Legend

No, you need to add that logic yourself. For file-based inputs, Splunk keeps track of where to start looking for new events in the file by storing to which position it's already read the file. You need to implement the same kind of thing in your scripted input yourself - find a unique incrementing ID of some kind, then after you've queried the REST webservice store the highest ID and then next time you're querying the webservice compare event ID's against that max ID to make sure you're only getting newer ones.

msn2507
Path Finder

I am redirecting the output of the webservice call to file and made it the source for splunk still I am seeing the duplicates. Note: I have checked the option "Follow tail"

Any help is appreciated

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

REGISTER NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If ...

Observability | Use Synthetic Monitoring for Website Metadata Verification

If you are on Splunk Observability Cloud, you may already have Synthetic Monitoringin your observability ...

More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk

Tuesday, May 14, 2024  |  11AM PT / 2PM ET Register to Attend Join us for this Tech Talk and learn how to ...