All Apps and Add-ons

Splunk add-on for opsec (4.0): 2-3 times as much data indexed as version 3

hcpr
Path Finder

Hi.
I just replaced the old opsec app with the new one (4.0) on our splunk system.
It works fine, but the amount of data indexed is significantly more than the old version from the same source.
I'm looking at 2 or 3 times as much data.
The amount and size of logs on the Checkpoint CLM has not increased. Actually I stopped logging several high-volume rules to try to reduce the amount of logs.

This particular firewall used to log 20-30 GB per day, now it's about 50-80 GB/day.

Doing som digging in the old and new data shows logs from the same CP products (appcontrol, url filter etc.) in the same proportions. So this is probably not a singe product suddenly logging more.

I have not been able to find any duplicates (which was my first thought) but I'm not sure the best way to do a search for duplicates. Any suggestions?

So has anyone else seen something similar? Any suggestions would be appreciated.

0 Karma

marios_kstone
Path Finder

We have a very similar issue on OPSEC LEA 4.2.0 and Splunk Core 6.6.0.
Actually, events are logged 13 times and this is obviously destroying our licenses. An easy query to detect the issue is:

sourcetype=opsec*|stats count, dc(_raw) as dedup by host|eval ratio=count/dedup

Strange thing is that a couple of weeks ago, events were indexed 9 times and now 13 times, which means that things get worse over time. Restarting splunk, resetting/recreating OPSEC inputs did not help either.
We have an open case with Splunk...

0 Karma

avisram
Path Finder

I'm having the same issue. Were you able to determine if/why you were getting duplicates?

0 Karma

gjanders
SplunkTrust
SplunkTrust

I would suggest you log a case with Splunk support as this is a Splunk supported application.

In addition you can set the application to "DEBUG" level logging which will provide more output/debugging information in your $SPLUNK_HOME/var/log/splunk/ (the log names include the keyword "checkpoint").
Alternatively, the command it runs can be run from the command line and you can see the raw output that the LEA lo grabber process returns, that might assist you.

Finally, 4.1.0 is out so perhaps you might want to try the latest version ?

0 Karma

hcpr
Path Finder

Iforgot to mention. This is 4.1.0 on Splunk 6.5

0 Karma

hcpr
Path Finder

Maybe I shouldn't answer myself, but since this i a rather large post I'll take a chance.

After some digging, its seems like Splunk or the LEA system actually records the logs from Checkpoint three times.

Here is a log record from Checkpoint Tracker, I've removed addresses etc.:
CP logdetail

And the same record when exported with fw log ( I had to slightly modify the format):

31Oct2016 10:00:00 allow removed-ip eth2-01.111 src:removed-ip;dst:193.212.4.120;proto:tcp;appi_name:Google Keep;app_desc:Google Keep is a note-taking service. Google Keep allows to write notes and color code them, to take pictures and record your voice. Supported from: R75.40.;app_id:60460461;app_category:Business Applications;matched_category:Business Applications;app_properties:SSL Protocol, Very Low Risk, Business Applications;app_risk:1;app_rule_id:{0EF98E02-F296-4AAB-AB87-F618F933362F};app_rule_name:default-ut;web_client_type:Other: Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko;app_sig_id:60460461:6;resource:http://clients4.google.com/invalidation/lcs/request;proxy_src_ip:removed-ip;bytes:2357;sent_bytes:19... logs:1;Referrer_self_uid:{58170810-0000-0000-8496-39100C0000C0};user:fjernet;src_user_name:fjernet;snid:51bfa9cd;product:Application Control;service:80;s_port:38975;product_family:Network

While in Splunk the same log looks like this:
Splunk log

The records in Splunk look the same, except for the "loc=" part and the fact that they are cut off at different points. The middle one seems to be the correct one in this sample, while the top and bottom ones are missing data.

Does anyone have a suggestion to where to start looking to figure this out?

Thanks.

0 Karma

georgen_splunk
Splunk Employee
Splunk Employee

the loc field is a sequential number and assigned to each event logged in the FW database. I'd first compare the events in Check Point to determine if they are actually the same. Would this also happen to be a clustered environment of some sort? ie host=wi_cluster Assuming this could replicate the same event to 3 nodes or separate events?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...