Splunk Dev

Annoymising IP but have a unique value for each IP

PeterLai1
New Member

Hi

We want to annoymise IPs, so far we can get it to replace all IP with x.x.x.x BUT we want to replace the IP with a unique value for each IP, so that we can see how many unique visitors and look up what they were doing without seeing any customer information.

Ideally, we want to do something like for (?m)^(.)clientip=\d+.\d+.\d+.\d+ (.)$ | sha256sum
REGEX = (?m)^(.)clientip=\d+.\d+.\d+.\d+ (.)$
FORMAT =$1 sha256sum the IP
DEST_KEY = _raw

Tags (1)
0 Karma

Damien_Dallimor
Ultra Champion

How is the data getting into Splunk ? If it is being sent by TCP/UDP/HTTP ... then my PDI App's powerful preprocessor framework may be able to help , some examples here of pre processing , http://www.baboonbones.com/blog/get-binary-data-splunk/

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi PeterLai1,
we had a similar problem and, following a Splunk people hint, we pre-parsed logs with an external batch procedure in PHP, encrypting one field with an external certificate.
In this way we anonymized field value and at the same time we're able in every moment to uniquely identify each value of the encrypted field with the reverse procedure.

Bye.
Giuseppe

0 Karma

PeterLai1
New Member

Hey, thanks for reply,

I got confused, as in your constantly scanning the file and update the encrypt the IP? How did you ensure that splunk doesn't get the pre-encrypted file?

Or as in you encrypted the access log and ourput to access_encrypted.log and splunk then monitors the access_encrypted.log file?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi PeterLai1,
we use the following process:

  • we receive logs from syslog,
  • we write them in a file (outside Splunk) named with date and time,
  • we encrypt the requested field,
  • we ingest parsed file in Splunk.

I understand that it isn't an easy process but this is the only way to solve our requirement.

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...