Getting Data In

Data masking using heavy forwarders

Venkat_16
Contributor

Been trying to mask data before indexing into indexer using heavy forwarders. below is the log sample and data am trying to mask

JSESSIONID=SD1SL10FF3ADFF3" to JSESSIONID=#######FF3ADFF3"

189.222.1.46 - - [24/Jul/2014:11:27:00] "GET /flower_store/product.screen?product_id=RP-SN-01 HTTP/1.1" 200 10897 "http://mystore.splunk.com/flower_store/category.screen?category_id=BALLOONS&JSESSIONID=SD1SL10FF3ADF..." "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10" 527 3006

10.2.91.38 - - [24/Jul/2014:11:28:00] "POST /flower_store/j_signon_check HTTP/1.1" 302 309
"http://mystore.splunk.com/flower_store/enter_order_information.screen&JSESSIONID=SD1SL10FF3ADFF3" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10" 3441 2576

192.0.1.38 - - [24/Jul/2014:11:28:15] "GET /flower_store/images/cat3.gif HTTP/1.1" 200 5024 "http://mystore.splunk.com/flower_store/item.screen?item_id=EST-21&JSESSIONID=SD1SL10FF3ADFF3" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10" 4323 3071

below is my props.conf and transforms.conf:

[access_common]
TRANSFORMS-anonymize = session-anonymizer

[session-anonymizer]
REGEX = (?m)^(.)JSESSIONID=\w{2}\d\w{2}\d{2}(\w+.)
FORMAT = $1JSESSIONID=#######$2
DEST_KEY = _raw

Kindly advice....i do not see and changes in fields after applying this configuration

0 Karma

woodcock
Esteemed Legend

This works:

| makeresults 
| eval raw="189.222.1.46 - - [24/Jul/2014:11:27:00] \"GET /flower_store/product.screen?product_id=RP-SN-01 HTTP/1.1\" 200 10897 \"http://mystore.splunk.com/flower_store/category.screen?category_id=BALLOONS&JSESSIONID=SD1SL10FF3ADFF3\" \"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10\" 527 3006:::10.2.91.38 - - [24/Jul/2014:11:28:00] \"POST /flower_store/j_signon_check HTTP/1.1\" 302 309 
\"http://mystore.splunk.com/flower_store/enter_order_information.screen&JSESSIONID=SD1SL10FF3ADFF3\" \"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10\" 3441 2576:::192.0.1.38 - - [24/Jul/2014:11:28:15] \"GET /flower_store/images/cat3.gif HTTP/1.1\" 200 5024 \"http://mystore.splunk.com/flower_store/item.screen?item_id=EST-21&JSESSIONID=SD1SL10FF3ADFF3\" \"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10\" 4323 3071"
| makemv delim=":::" raw
| mvexpand raw
| rename raw AS _raw
| rex mode=sed "s/JSESSIONID=\w{2}\d\w{2}\d{2}(\w+.)/JSESSIONID=#######\1/"

So you can use this:

 SEDCMD-session-anonymizer = s/JSESSIONID=\w{2}\d\w{2}\d{2}(\w+.)/JSESSIONID=#######\1/
0 Karma

cpetterborg
SplunkTrust
SplunkTrust

All you need in this case is the following line in your props.conf file (no transforms.conf config needed) for the sourcetype:

SEDCMD-hidesessionid = s/JSESSIONID=\w{2}\d\w{2}\d{2}/JSESSIONID=#######/
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @Venkat_16,

Please use below configuration in transforms.conf on Heavy forwarder.

[session-anonymizer]
REGEX = (?m)^(.*)JSESSIONID=.*((?=\"\s\").*)$
FORMAT = $1JSESSIONID=#######$2
DEST_KEY = _raw

After changing above configuration please reload configuration using https://<HF FQDN>:8000/debug/refresh OR restart splunk on heavy forwarder.

I hope this helps.

Thanks,
Harshil

Venkat_16
Contributor

Thanks for your answer @harsmarvania57. But it didnt work 😞

[access_common]
TRANSFORMS-anonymize = session-anonymizer

[session-anonymizer]
REGEX = (?m)^(.)JSESSIONID=\w{2}\d\w{2}\d{2}(\w+.)
FORMAT = $1JSESSIONID=#######$2
DEST_KEY = _raw

i restarted heavy forwarder also....logs are getting indexed with out any masking

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Looks like you are still using your old configuration, please use configuration which I have provided.

0 Karma

Venkat_16
Contributor

sorry it was copy paste error:

[session-anonymizer]
REGEX = (?m)^(.)JSESSIONID=.((?=\"\s\").*)$
FORMAT = $1JSESSIONID=#######$2
DEST_KEY = _raw
is the one am using

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Still this configuration is wrong, you can see my regex has (?m)^(.*)
and configuration which you pasted starts with (?m)^(.)

0 Karma

Venkat_16
Contributor

Thanks alot for helping patiently but still it didnt work:

[access_common]
TRANSFORMS-anonymize = session-anonymizer

[session-anonymizer]
REGEX = (?m)^(.)JSESSIONID=.((?=\"\s\").*)$
FORMAT = $1JSESSIONID=#######$2
DEST_KEY = _raw

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Please refer this https://regex101.com/r/CftIqK/1, regex is working perfectly fine. Can you please provide your configuration in Code Sample format because your REGEX is still wrong, you can see 101010 button when you type comment/answer please use that and paste your transforms.conf configuration.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...