All Apps and Add-ons

Can Splunk DB Connect reformat data before indexing?

ankithreddy777
Contributor

After we retrieve the data from the database, can Splunk DB Connect reformat data before indexing?

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

@ankithreddy777 - Were you able to test out nabeel652's solution? Did it work? If yes, please don't forget to resolve this post by clicking on "Accept". If you still need more help, please provide a comment with some feedback. Thanks!

0 Karma

nabeel652
Builder

You can do all sorts of reformatting through the SQL provided in the Inputs in DB Connect. So change all the formats like timestamp, field values based on case statement etc while providing the query in database inputs.

Below is my query that I use for getting SCOM ACS logs form the SCOM ACS DB through DB Connect. Note all sorts of data formatting done through convert and case commands:

SELECT 
 convert(varchar,dvA.CreationTime,120) + ' +0000' AS CreationTimeUTC,
 dvA.EventId,
 dvA.EventMachine AS Server,
 dvA.TargetUser AS Username,
CASE dvA.ClientDomain
   WHEN 'n/a' THEN PrimaryDomain
   ELSE dvA.ClientDomain
 END AS Domain,
 CASE dvA.String02
   WHEN 'n/a' THEN NULL
   ELSE dvA.String02
 END AS SourceMachine,
 dvA.CollectionTime
FROM
 AdtServer.dvAll5 AS dvA
WHERE
 dvA.EventId IN (4624,4648,4672,4634,4737,4740,4625,4767,4771,4738,644) AND CreationTime > convert(datetime, '2016-10-03 16:00:37.207')
0 Karma

langlv
Engager

I have same situation. There are fields in database which contain sensitive data and I dont want to show the clear text in Splunk. Instead, I am going to transform/encrypt those fields data before sending to Splunk for indexing.

 

Currently I am using DB Connect 2.x, and manually modify the code of the app (Python) to encrypt data before writing event to indexing stream. But since DB Connect 3.x I'm not able to do that.

 

Is there any way that I can add the custom handler for data stream from database, process it before return to DBX to submit to Splunk HEC ?

 

Thanks,

Lang

0 Karma

langlv
Engager

Be noted that I don't want to use any custom modification in database due to performance impact. I tried to apply encryption on fields in select query and it turned out very high CPU in database.

 

By moving data encryption to Splunk DBX, I can scale out the work load to a cluster of heavy forwarders.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...