I had the same issue. Setting the interval correctly is insufficient. There is too much room for error to either duplicate or miss a record. I needed a more solid solution. I changed the query in inputs.conf to call a function that will handle the duplicate checking. That way if there is overlap it can be handled by the function.
This is based on using DB Connect v2 with postgres but should work for mysql as well.
In input.conf, change from:
query = INSERT INTO <fully qualified table name>(_field1,_field2,_field3) VALUES (?,?,?)
to
query = update <fully qualified table name> set <_field1>=<_field1> where 1=(SELECT <fully qualified function name>(?,?,?))
Sadly, I found the update wrapper necessary because the db output routine expects to be calling INSERT and expects no result. If you call SELECT directly, an empty result set is returned by the function that cause db output to throw an error
I've read in other posts that using the DB Connect gui may overwrite the input.conf. I have not experienced this yet, but I would watch out for that.
Also, once the edit is made, restart DB Connect with:
sudo su - splunk
cd /opt/splunk/bin
./splunk _internal call /services/apps/local/splunk_app_db_connect/_reload -auth $ADMIN_LOGIN:$ADMIN_PWD
... View more