Splunk Enterprise Security

Why is a Splunk Enterprise Security Threatlist failing to download with error "Bandwidth Limit Exceeded"?

jwelch_splunk
Splunk Employee
Splunk Employee

We are seeing this error:

2015-12-16 08:02:56,545 ERROR pid=42684 tid=MainThread file=protocols.py:run:226 | Caught HTTPError when querying http://data.phishtank.com/data/online-valid.csv.gz: code=509 exc=HTTP Error 509: Bandwidth Limit Exceeded
0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

Splunk Enterprise Security App provides several pre-configured Threat Intelligence download sites in the OOB configuration that are avilable for you to enable and use.

These sites are operated / maintained by organizations outside of Splunk. These vendors have some limitations to their free offerings.

This error is one example of that: Phishtank limits access to this free service to 75 connections in a rolling 72 hour period.

At present Threat Intelligence downloads when enabled are done on the ES SH, and done every 12 hours. (configurable)

So one can infer that 1 ES SH would touch Phish Tank 2 times a day, but because we download the Threat Intelligence on each SH, if you are running an 5 node SHC for ES, this would grow 10 connections a day. Of course this excludes system restarts which can also trigger a download.

Customers that are PAT'ing/SNAT'ing their hosts leaving for the internet might have other systems in the Enterprise that also use these free services which would appear to Phishtank as all coming from the same system.

So it easy to see how this can become an issue.

Solution Possibilities:
1. Pay for a subscription service with these vendors and often times the connection limit will be removed.
2. Run a search on your Splunk Servers looking at your firewall data ( = and see if other hosts are also destined for those same destination addresses, if they are and you are hiding all hosts behind a firewall as mentioned earlier work with the other admins to tune down how often you are reaching out, as Phishtank would see you all to be the same source address.
3. Tune down the frequency on your SHC Nodes so you are not hitting the limits.

Okie

asimagu
Builder

hi Okie @jwelch
if you register with Phishtank you can register an app and they give you a key (a long alphanumeric chain) to remove the limit. Is there any way in Splunk of configuring that key for the threatlist download?

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

Is it a URL modification? Is it a userid/password (auth) combo?

I would assume it would have to be tied to one of these. And both of those should be configurable.

If you want to send me what they sent you I can take a look at it.

okie at splunk / com

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...