About langlv

langlv · ‎06-10-2020

Hi ninjas, I am using DB Connect 2.x for getting data from DB to Splunk. There are some sensitive fields which are not allowed to show in clear text, hence I had to hash/encrypt the data before indexing in Splunk. I tried to hash/encrypt the fields in SQL, but it turned out very high CPU consumption in DB. I solved this issue by modified DB Connect 2.x code (in Python) to encrypt field data before sending to event stream. This also helped to scale out the computation to a cluster of heavy forwarders. But with DB Connect 3.x I am unable to do that. Are there any solution to hash/encrypt the field data before indexing to Splunk using DB Connect 3.x ? Something like adding a custom handler to process the data/result set from DB before DBX 3.x sending the events to HEC. I am going to upgrade to DBX 3.x because of its performance and stability. I found the same requirement in this post but no solution yet (https://answers.splunk.com/answers/488681/can-splunk-db-connect-reformat-data-before-indexin.html) Thank you very much. Lang

langlv · ‎06-10-2020

Be noted that I don't want to use any custom modification in database due to performance impact. I tried to apply encryption on fields in select query and it turned out very high CPU in database. By moving data encryption to Splunk DBX, I can scale out the work load to a cluster of heavy forwarders.

langlv · ‎06-10-2020

I have same situation. There are fields in database which contain sensitive data and I dont want to show the clear text in Splunk. Instead, I am going to transform/encrypt those fields data before sending to Splunk for indexing. Currently I am using DB Connect 2.x, and manually modify the code of the app (Python) to encrypt data before writing event to indexing stream. But since DB Connect 3.x I'm not able to do that. Is there any way that I can add the custom handler for data stream from database, process it before return to DBX to submit to Splunk HEC ? Thanks, Lang

langlv · ‎07-27-2017

Thanks vasanthmss, combining autoregress with streamstats and eventstats is what i am looking for. It works.

langlv · ‎07-13-2017

Thanks vasanthmss for your help, but it still count all INQ by ATM1 while I need only consecutive INQ on ATM1. If another CWD on ATM1 occur the counting will be reset. Pls consider following sample events: terminal_id tran_type ATM1 INQ ATM2 CWD ATM1 INQ ATM1 INQ ATM2 CWD ATM1 INQ ATM2 INQ ATM1 CWD ATM3 INQ ATM3 INQ ATM1 CWD ATM3 INQ ATM3 CWD ATM1 INQ The last ATM1,INQ should be be counted because there is ATM1,CWD happen before. Thanks,

langlv · ‎07-13-2017

Eventstats also does not help because it count all the INQ by ATM1 while I need only consecutive events. If I add 1 more ATM1,INQ to the event list like this: | makeresults | eval input="ATM1,INQ-ATM2,CWD-ATM1,INQ-ATM1,INQ-ATM2,CWD-ATM1,INQ-ATM2,INQ-ATM1,CWD-ATM3,INQ-ATM3,INQ-ATM1,CWD-ATM3,INQ-ATM1,INQ" | makemv delim="-" input | mvexpand input | rex field=input "(?[^,]+),(?.+)$" | eventstats count by terminal_id, tran_type | where count >= 3| table terminal_id, tran_type Then the last INQ event should not be counted because there is another CWD on ATM1 before INQ happen.

langlv · ‎07-13-2017

Thanks for your comment, but | stats count... does not meet the business requirement as I need to count only consecutive INQ on terminal. If there is CWD on ATM1 then the INQ counting will be reset. I also really to list out the events for business cross check.

langlv · ‎07-13-2017

Hello, I am trying to find out the list of consecutive card transactions on same terminal in period of time, eg: more than 3 inquiry transactions in a ATM machine from the hold transaction history of the system (from multiple ATMs). terminal_id tran_type ATM1 INQ ATM2 CWD ATM1 INQ ATM1 INQ ATM2 CWD ATM1 INQ ATM2 INQ ATM1 CWD ATM3 INQ ATM3 INQ ATM1 CWD ATM3 INQ ATM3 CWD ATM1 INQ Given the parameter for listing consecutive INQ transactions on same terminal is 3, I want to list out the following: terminal_id tran_type ATM1 INQ ATM1 INQ ATM1 INQ ATM1 INQ ATM3 INQ ATM3 INQ ATM3 INQ where both ATM1 & ATM3 need to be checked (ATM1 has 4 and ATM3 has 3 INQ consecutive transactions individually). Thank you guys for your help. Lang

langlv · ‎02-09-2017

It works like a charm, add some isnull() check and i can now have balance for all accounts. Thanks ehudb.

langlv · ‎02-09-2017

Hi, I have a list of events here from an account management system. "_time",dr_account,cr_account,amount "2017-02-09T17:20:19.000+0700",111111,222222,1500000 "2017-02-09T17:20:19.000+0700",111111,333333,1000000 "2017-02-09T17:20:19.000+0700",222222,555555,11070000 "2017-02-09T17:20:19.000+0700",333333,111111,3000000 "2017-02-09T17:20:19.000+0700",444444,111111,1250000 "2017-02-09T17:20:19.000+0700",111111,555555,3000000 I am going to make a aggregation report by sum the amount field by both dr_account and cr_account and, then display the total debit, total credit and balance (=total debit - total credit) on a table for each account, for example: account,total_debit,total_credit,balance 111111,5500000,4250000,1250000 222222,11070000,1500000,9570000 ... I am trying to use stats sum(amount) as total_debit by dr_account | appendcols [stats sum(amount) as total_credit by cr_account] but no luck. Could you guys please help me on this case ? Thank you very much. Lang

langlv · ‎05-31-2016

It works like a charm. Thanks Woodcock,

langlv · ‎05-30-2016

Hi pros, I am new with Splunk and trying to analyze a complex log file from a financial application. I want to figure out the fields from a multilines event, here is my log example: 16.02.10 09:20:53 [ FromIso:123456789]************** INBOUND MESSAGE ID[AAABqgAwV0ujhQAA] *************** in[ 48: ]<800> in[ 48: ]<8220000100000000> in[ 48: ]<0400000000000000> msgno[ 0]<800> Bitmap: [82200001000000000400000000000000] in[ 7: ]<530> in[ 7: ]<92833> in[ 11: ]<694437> in[ 32: ]<6> in[ 32: ]<123123> in[ 70: ]<2> D-ISO-0306: m0800/a0000000000/t694437/p000000/r00 16.02.10 09:20:53 [ ToIso:123456789]************** OUTBOUND MESSAGE ID[AAABqgAwV0ujhQAA] *************** msgno[ 0]<810> Bitmap: [82200001020000000400000000000000] out[ 48: ]<0810> out[ 48: ]<8220000102000000> out[ 48: ]<0400000000000000> out[ 7: ]<0530> out[ 7: ]<092833> out[ 11: ]<694437> out[ 32: ]<06> out[ 32: ]<123123> out[ 39: ]<00> out[ 70: ]<002> I want to extract the fields msgno=800/810 and field39="00" from the log above. I tried to use Field Extractor to extract those fields but no luck. Thanks, Lang

Posts	12
Solutions	0
Karma Given	1
Karma Received	0
Member Since	‎05-29-2016

Online Status	Offline
Date Last Visited	‎07-02-2020 05:57 AM

DB Connect 3: reformat/encrypt fields data before ...

Find the list of consecutive events by a field

Sum field value by different fields and merge toge...

Extract field from a complex multi-lines event fro...

DB Connect 3: reformat/encrypt fields data before ...

Re: Can Splunk DB Connect reformat data before ind...

Re: Can Splunk DB Connect reformat data before ind...

Re: Find the list of consecutive events by a field

Re: Find the list of consecutive events by a field

Re: Find the list of consecutive events by a field

Re: Find the list of consecutive events by a field

Find the list of consecutive events by a field

Re: Sum field value by different fields and merge ...

Sum field value by different fields and merge toge...

Re: Extract field from a complex multi-lines event...

Extract field from a complex multi-lines event fro...