Unable to mask PII data on JSON at indexer

[mehala12]

Hello friends,
My data is in json format and i have credit card info which i need to mask at indexer level. I tried below regex at props.conf but the masking is not happening:

SEDCMD-ccmask = s/\b(?:[0-9]{11}(\d{4}))(\b)/XXXMASKEDXXX\1/g

When i run the same command at search head, it shows result as XXXMASKEDXXX [index=foo sourcetype=bar
| rex field=_raw mode=sed "s/\b(?:[0-9]{11}(\d{4}))(\b)/XXXMASKEDXXX\1/g"]. But the same command doesnt work when applied at indexer (please note that the indexer was restarted after update)

Please help me to understand the issue and let me know how to mask data which is in json format.

[woodcock]

You need to deploy your change to ALL indexers and restart ALL splunk instances running there. When testing, only look at events that have been indexed AFTER the restarts (older events will stay as they were when they were indexed).

[mehala12]

Hello woodcock, yes, mine is a standalone indexer and i restarted after the change. Also, i was looking at newly indexed data for output.

[woodcock]

Are you SURE that your sourcetypes (or host, if that is what you are using in your stanza header in props.conf) match? Are you using sourcetype rename?

[mehala12]

Ok, here is the thing. My data is forwarded by a different team which i don't have access to. I need to check how props is configured there. Thanks for the heads-up.

[DalJeanis]

1) First, the length of a credit card (Visa, MC and many others) is normally 16 digits. American Express credit cards, Japan Credit Bureau debit cards, and Maestro debit cards are the only issuers that are still using 15. If this is a general use masking routine, then that {11} needs to be {12} and/or the whole rex/sed needs to be slightly more complicated to deal with various debit cards that range from 12 to 19 digits.

2) I'm not sure why you use [0-9] for one set of digits and \d for the other, since they are equivalent and the latter is more concise.

3) I'm not sure why you've marked the second word boundary as a capture group and then didn't use it.

So, here's a run-anywhere sample showing the simplest rex that does work for 16 digit cards.

| makeresults 
| eval cc1="1234567890123456" 
| rename COMMENT as "The above just generates test data."

| rename COMMENT as "The below shows before and after length, and the masked data."
| eval lenbefore=len(cc1) 
| rex mode=sed field=cc1 "s/\b(?:\d{12}(\d{4}))\b/XXXMASKEDXXX\1/g" 
| eval lenafter=len(cc1)

Here's a version that masks all credit cards from 12 to 19 digits. However, it has the side effect of leaving them all sized at 16 digits, which may or may not be appropriate.

| makeresults 
| eval cc1="123456789012 1234567890123 12345678901234 123456789012345 1234567890123456 12345678901234567 123456789012345678 1234567890123456789" 
| makemv cc1 
| mvexpand cc1
| rename COMMENT as "The above just generates test data of each possible length."

| rename COMMENT as "The below shows before and after length, and the masked data."
| eval lenbefore=len(cc1) 
| rex mode=sed field=cc1 "s/\b(?:[0-9]{8,15}(\d{4}))(\b)/XXXMASKEDXXX\1/g" 
| eval lenafter=len(cc1)

This version might be useful if you need to be able to track back what KIND of card was used in any particular case, so the first digit should not be masked...

| makeresults 
| eval cc1="123456789012 22345678901234 32345678901234 423456789012345 5234567890123456 62345678901234567 723456789012345678 8234567890123456789" 
| makemv cc1 
| mvexpand cc1 
| rename COMMENT as "The above just generates test data of each possible length."

| rename COMMENT as "The below shows before and after length, and the masked data."
| eval lenbefore=len(cc1) 
| rex mode=sed field=cc1 "s/\b(\d)(?:[0-9]{7,14}(\d{4}))(\b)/\1XXXMASKEDXX\2/g" 
| eval lenafter=len(cc1)

[mehala12]

Thanks daljeanis, the examples work well at search head but it doesn't work on indexer. My card length is 15 digits long.