I am using the REST TA ( https://apps.splunk.com/apps/id/rest_ta ) to pull data from an API which outputs CSV data. The API allows me to pull all events or last 10 events and since I need everything, I need to pull all every time. This means that there is duplicated events every time the REST TA pulls data.
I need to use a key of some sort to avoid duplicates and do not know where to start. A search on this answers board and on stackoverflow are not resulting in answers that are what I need.
Is there a way to manually specific the ID/Key used to index the data? If so, then that would presumably prevent duplicates since it cannot be duplicated. Or, what I am doing on elasticsearch, is using a duplicate ID to overwrite existing data in the index that has the same key with the new data. That is also a possibility as some of the data from the source could have changed but needs to be updated (like if the issue is pending or resolved, etc.)
Thanks in advance.
... View more