/opt/splunk/etc/apps/sophos_central/bin/sophos_events.py is pulling down many duplicate events

[ehoward]

This script is pulling down duplicate events eveytime it makes an API connection. My index is getting clogged with a massive backlog of duplicate events and I am 24 hours behind in my logs because to the retrieval batch limit of 1000 in the API. I am not sure if this is an issue with the actual script or if there is an issue with API event retrieval.

[ehoward · Accepted solution]

After much troubleshooting this appears to be an issue with the Sophos API. We use Sophos WebControl and the API seems to returning an insane amount of duplicate WebControl events. I am going to open a ticket with Sophos.

[nickhills]

Are you by chance running this on Windows? - In which case I think I may know what the issue is.

[ehoward]

Alas, no. My Splunk server is running Debian Linux.

[nickhills]

any reason the splunk user would not be able to write to the temp folder?
The 1.0.2 has a hardcoded (whoopsie) path to /tmp to write the cursor file - if it was unable to write this file it would re-read the last 24hours events every time the import runs.

[ehoward]

The cursor file in /tmp gets re-written every time the script runs. I can see the timestamp change and I have confirmed that the value changes. Is there a chance that there is an issue with pulling out the correct cursor value when the script runs? More importantly, are you seeing the dupe issue in your own environment?

[nickhills]

no, there are no dupes in my environment (fedora),
but I have a new build which includes a couple of other fixes. i'm just smoke testing it, then will upload it to Splunk base

[ehoward]

Well, I'm foxed. I'm going to revert to the Sophos scripts for now. Maybe I can normalize the data later so your App can display it.

[ehoward]

I reverted to the Sophos API scripts. It looks like Sophos WebControl events get repeated like crazy when they are pulled down via the API. I am going to modify your Python script to use a Python dictionary to dump dupes.

Actually, I need to open a support ticket with Sophos. There are so many repeat events that the damn cursor can not keep up due to the 1000 event API retrieval lmit.

Thank you for responding to my questions.

[ehoward]

I may disable your App and revert to using the sample scripts in the Sophos API reference documentation with a cron job to pull down the events in JSON format and see if I get any dupes. That will allow me to determine if there is something flaky with my API key.

Assuming that works is there any chance of recoding your App to use the Sophos retrieval scripts to pull in the KV values?

[ehoward]

It appears that when the script pulls down new events from the API those same events they keep getting downloaded every subsequent run. I am running a Realtime search for the newest EventId (fieldname 'id)' that popped up in and I get getting the same event being added into my Splunk index every time the script hits its script execution interval as defined in inputs.conf.

[ehoward]

The following queries show that I am getting massive amounts of dupes from this script

source="/opt/splunk/etc/apps/sophos_central/bin/sophos_events.py" | stats count values(host) values(source) values(sourcetype) values(index) by _raw | WHERE count>1

search source="/opt/splunk/etc/apps/sophos_central/bin/sophos_events.py" | eval dupfield=_raw | transaction dupfield maxspan=1s keepevicted=true

I don't know if it is your script or something is foobared with my API key.

[ehoward]

I assume that you also use Sophos Cloud. Can you replicate my dupe issue in your environment using the above searches?

[ehoward]

Nick, are you affiliated with Sophos?

[nickhills]

As per my comment in your other question:

No - in no way whatsoever.
Just a Splunk user who shared his efforts with the community.

[ehoward]

Well, I appreciate the effort you put into the App. I was wondering why you decided to create your own sophos_events.py log retrieval script for your App instead of using the official Sophos script at (https://github.com/sophos/Sophos-Central-SIEM-Integration. ) I wonder if there is something with the eventCursor (or my API key) that is screwed up.

[nickhills]

Principally it was to retrieve the data in simple k=v pairs so no further work was needed at index time, it's also a massive simplification over the Sophos version which used to output in CEF format (although it looks like they have recently updated it to support JSON and KV too, however when I published the app they were not options!)

[ehoward]

I like the App. I wish I knew whether the duping issue was unique to my environment (or API key)

[denose]

I am experiencing the same duplication of events, when it gets events, seems to have stopped since the first time that it scraped in all the previous events.

[ehoward]

denrose, this issue is now known to Sophos. I have a ticket in with the Sophos Global escalation team to get this resolved. Sophos appears to be storing multiple duplicate events in Sophos Cloud logs. For reasons I cannot even fathom they do not appear to to using any sort of Key field in their log data store to prevent duplicate entries from being recorded.

[nickhills]

When the script runs it fetches everything which the Sophos API returns ( which I think is a few days) Once the script has caught up it should only fetch new events.

There are a couple of cases where this could go wrong - if you were running on windows prior to 1.0.5, or python cant write a temporary file it may have struggle to write a checkpoint file, thereby re-importing the same events over and over. However If you are on the latest version and still having issues, it sounds like there maybe another glitch, or the Sophos API as ehoward mentioned - although I must say that feels less likely.