Deployment Architecture

dbconnect - data base input (dump) - double events every time it runs

cramasta
Builder

I have a db input setup to take a dump using a query once a day.

My settings are
-Dump
-I have a custom query
-Key-Value Format
-Include Timestamp

Everytime that the input runs i get duplicates of each event. My query returns results that dont contain a timestamp which is why I configured the input to create one. Each duplicate event will have the same splunk generated timestamp.

If i run the same query with the dbquery command i get the correct number of results

Any ideas why this is happening?

0 Karma

ziegfried
Influencer

Have you upgraded from an older version of DB Connect/DBX? If so, which version?

0 Karma

Dan
Splunk Employee
Splunk Employee

I believe that is expected behavior for the dump command. The full results of the query will be indexed every time. If you don't have a suitable rising column in the table, you will not be able to get just the new events.

Can you please post your custom query, or better yet, the contents of inputs.conf?

Thanks!

cramasta
Builder

Hi Dan.

The thing is I have the dump setup as a cron job that runs once a day. If I clear the index and wait till the next time the job runs I find duplicate events for each row returned from the query. Im letting splunk generate the index time for each event and I am finding that each duplicate event has the same index time. I would expect to see different timestamps if it was from a previous dump. I found with troubleshooting that setting the input to use the table name instead of the query will only index the table data x1. Ill add my inputs.conf and query shortly.

Thanks,
Joe

0 Karma

cramasta
Builder

FYI - If i tell it to dump the table instead of a query it doesn't index events x2

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...