Getting Data In

After deploying an app with a sedcmd stanza in props.conf, why is my data not being anonymized?

Conradj
Path Finder

Hi,

I want to anonymize sessionid information from weblogs =.

I use a deployment server to push out an app with the log files we are tailing.

In that app, I have a props.conf with the following line:

[web_access]
SEDCMD-access = s/(?:\s\d+\s)(\w{32})/ XXXXXXXXXX-sessionid-XXXXXXXXXXX /g

web_access is the sourcetype of the log being tailed that contains the session id.

The session id (char length 32) is always preceded by an integer surrounded by white space.

I came to the regex above by tweaking the results of a search with rex mode=sed "s/(?:\s\d+\s)(\w{32})/XXXXXX-sessionid-XXXXXX/g. This consistently masks the sessionid in searches on historical data.

I have deployed the the app out with its new sedcmd stanza in props.conf, but new data doesn't seem to be getting anonymized, even though I have restarted the universal forwarder on the web server (Windows, but not iis)

Any ideas?

0 Karma
1 Solution

Conradj
Path Finder

after some tweaking it appears the props.conf must exist on the indexer rather than the universal forwarder.

I had a copy of props.conf on the indexer, i removed it and the masking broke.

edit:

After some monitoring and further tweaking, this is the final sedcmd we are using:

[web_access]
SEDCMD-access = s/(\s\d+\s)(\w{16})(\w{16})/ \1 XXX-sessionid-XX\3 /g

The inital sedcmd also took out the integer surrounded by white space (the response size), so this puts it back. And we also only mask half of the sessionid. As after consultation with our devs and infosec it was determined that having half the session id meant they could still use it to trouble shoot user flows, and because we had masked half of it it means the user session cannot be stolen for a replay attack. half of the sessionid should still be unique over short intervals that we would need to trouble shoot user flows.

The replacement anonymized sessionid is also 32chars, so it maintains formatting in the log (if that was ever a problem anyway)

View solution in original post

0 Karma

Conradj
Path Finder

after some tweaking it appears the props.conf must exist on the indexer rather than the universal forwarder.

I had a copy of props.conf on the indexer, i removed it and the masking broke.

edit:

After some monitoring and further tweaking, this is the final sedcmd we are using:

[web_access]
SEDCMD-access = s/(\s\d+\s)(\w{16})(\w{16})/ \1 XXX-sessionid-XX\3 /g

The inital sedcmd also took out the integer surrounded by white space (the response size), so this puts it back. And we also only mask half of the sessionid. As after consultation with our devs and infosec it was determined that having half the session id meant they could still use it to trouble shoot user flows, and because we had masked half of it it means the user session cannot be stolen for a replay attack. half of the sessionid should still be unique over short intervals that we would need to trouble shoot user flows.

The replacement anonymized sessionid is also 32chars, so it maintains formatting in the log (if that was ever a problem anyway)

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

For posterity, I will add this link, which explains which settings take effect on which components of a deployment; in the hopes it is helpful.

0 Karma

Conradj
Path Finder

yes, this is very useful thank you.

I have decided to retain the changes in the app to include the props.conf but to have a comment that the stanza needs to exist on the indexer. just in case the props.conf on the indexers i lost or overwritten, the app config will tell whoever how to get it working again.

jplumsdaine22
Influencer

Glad you figured out your own question! If possibl;e, could you accept your own answer to mark the question completed?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...