Getting Data In

Can Splunk Cloud filter out events without an on-prem Heavy Forwarder?

stefanovalentin
New Member

Is it possible to implement event filtering (and/or routing) in a managed Splunk Cloud deployment without the usage of an on-prem Heavy Forwarder?

The scenario is:
- Running a managed Splunk Cloud instance
- Need to ingest AWS Cloudtrail logs (preferably using the "AWS Add-on" app and configure a SQS-Based S3 input)
- Need to filter out the majority of Cloudtrail events before they hit the Index and so impact the license
- (Critical requirement) Need to avoid "external" self-managed components like Heavy Forwarders (main reason to get a managed Cloud instance)

If the answer is no, I am also open to other suggestions.

Note: I am aware that AWS Add-on allows to setup a "Generic S3" input for Cloudtrail that implements "in-line" event blacklisting. Unfortunately this is not an option in my scenario as the input type is way too heavy on S3 operations.

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Hi @stefanovalentino,

Yes, you can do that with below props and transforms.conf configuration. You need to put this on Indexer on Cloud instance. This configuration will stop the event from indexing on the indexer.

props.conf
[<source/sourcetype/host on which you want to filter the events>]
TRANSFORMS-filter_events = filter_events_tr

transforms.conf
[filter_events_tr]
REGEX = <regex which defined which events to be filtered out from _raw>
DEST_KEY = queue
FORMAT = nullQueue

Hope this helps!!!

stefanovalentin
New Member

Thank you @VatsalJagani !

Follow-up question - what's the best way to manage props/transform files in a Cloud instance? I have heard you can manage those through vetted apps, but the process of updating an app through Splunk support can be tedious and slow sometimes...

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@stefanovalentino - No, In Splunk cloud, the only way is that you ask the support team to put these configurations. As this requires backend access. Even if there would have been a way, I would have recommend that you go through Cloud support team.

0 Karma

ColinJacksonPS
Path Finder

I've been trying to figure this one out, but Splunk Cloud basically said to make a custom app to get this to work. Barebones app, AppInspect, Failed vetting. 
No app.conf, random /tmp/<blah blah> directory errors. It's a mess. 

tl;dr - you can't send them a conf file and say, "please install this"

Tags (1)
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...