Hey everyone,
I have an issue where I am ingesting data via REST API, though I am getting a lot of duplicate data into the index. It seems the issue resides on the table where the API sources from, so in the meantime I have to dedup the results.
index=index1 sourcetype=dataset1 | dedup data_id | table column_1, column_2, column_3
My question is, is there a way to run the dedup command within the props.conf file ?
I have read where I could do an eval =mvdedup(value) command, but I would need to dedup across the events and not just one field
Any thoughts?
... View more