All Apps and Add-ons

How to update database table rows with Splunk data?

matstap
Communicator

I want to update a database table with data from Splunk. I need to join on an ID field. So if my data has a row with ID = A and the database table has a row with ID = A, that row should be updated with my new data. If my row has an ID that isn't already in the database table, add the new row.

I've tried dbxoutput, but that seems to only append all the new data without joining on any field. How have others achieved this?

0 Karma
1 Solution

kheo_splunk
Splunk Employee
Splunk Employee

If you want to upsert functionality in a database, "Enable Upsert" should be checked with a proper column name(in your case - ID column) under "Upsert Configuration" section in the 3rd step(Fields Mapping) when creating an database output.

Please check the following Splunk doc.
http://docs.splunk.com/Documentation/DBX/3.1.3/DeployDBX/Createandmanagedatabaseoutputs#Fields_mappi...

I have tested this funtionality using EMPLOYEES table on Oracle database and it worked for me. After configuring the database output using DB Connect UI, $SPLUNK_HOME/etc/apps/splunk_app_db_connect/local/db_outputs.conf shows the following. Please note that EMPLOYEE_ID is configured for unique_key and using_upsert as 1 below:

db_outputs.conf
[EMP]
connection = oracle
customized_mappings = COMMISSION_PCT:COMMISSION_PCT:3,DEPARTMENT_ID:DEPARTMENT_ID:3,EMAIL:EMAIL:12,EMPLOYEE_ID:EMPLOYEE_ID:3,FIRST_NAME:FIRST_NAME:12,JOB_ID:JOB_ID:12,LAST_NAME:LAST_NAME:12,HIRE_DATE:HIRE_DATE:93,PHONE_NUMBER:PHONE_NUMBER:12,SALARY:SALARY:3,MANAGER_ID:MANAGER_ID:3
disabled = 0
is_saved_search = 0
scheduled = 0
search = index=main sourcetype=dbx | search EMPLOYEE_ID=173
table_name = "SPLUNK"."EMPLOYEES"
ui_query_schema = SPLUNK
ui_query_table = EMPLOYEES
unique_key = EMPLOYEE_ID
using_upsert = 1

View solution in original post

0 Karma

kheo_splunk
Splunk Employee
Splunk Employee

If you want to upsert functionality in a database, "Enable Upsert" should be checked with a proper column name(in your case - ID column) under "Upsert Configuration" section in the 3rd step(Fields Mapping) when creating an database output.

Please check the following Splunk doc.
http://docs.splunk.com/Documentation/DBX/3.1.3/DeployDBX/Createandmanagedatabaseoutputs#Fields_mappi...

I have tested this funtionality using EMPLOYEES table on Oracle database and it worked for me. After configuring the database output using DB Connect UI, $SPLUNK_HOME/etc/apps/splunk_app_db_connect/local/db_outputs.conf shows the following. Please note that EMPLOYEE_ID is configured for unique_key and using_upsert as 1 below:

db_outputs.conf
[EMP]
connection = oracle
customized_mappings = COMMISSION_PCT:COMMISSION_PCT:3,DEPARTMENT_ID:DEPARTMENT_ID:3,EMAIL:EMAIL:12,EMPLOYEE_ID:EMPLOYEE_ID:3,FIRST_NAME:FIRST_NAME:12,JOB_ID:JOB_ID:12,LAST_NAME:LAST_NAME:12,HIRE_DATE:HIRE_DATE:93,PHONE_NUMBER:PHONE_NUMBER:12,SALARY:SALARY:3,MANAGER_ID:MANAGER_ID:3
disabled = 0
is_saved_search = 0
scheduled = 0
search = index=main sourcetype=dbx | search EMPLOYEE_ID=173
table_name = "SPLUNK"."EMPLOYEES"
ui_query_schema = SPLUNK
ui_query_table = EMPLOYEES
unique_key = EMPLOYEE_ID
using_upsert = 1

0 Karma

mchang_splunk
Splunk Employee
Splunk Employee

You can have dbxquery to call store procedure to achieve.

http://docs.splunk.com/Documentation/DBX/3.1.3/DeployDBX/Commands#Required_Arguments

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...