All Apps and Add-ons

Index S3 logs

blankenhaus
New Member

Hello !

I am having trouble indexing only the S3 logs and not any other objects within a S3 bucket.
I am using Splunk Add-on for AWS Splunk_TA_aws 1.1.1.
I have a IAM role defined with access to my bucket and that seems to work fine.
I have also enabled logging for my bucket. That created a logs/ key within my bucket. All log files are within that key, for instance:
...
test-bucket/logs/2015-07-08-00-16-59-89A88B0EDD3514E2
...

However, when running a search all of the objects within the bucket are indexed.
Here is my config file:

$ cat /opt/splunk/etc/apps/Splunk_TA_aws/local/inputs.conf
[aws_s3://s3logtest2]
aws_account = test-account
blacklist = ^.$
bucket_name = test-bucket
character_set = auto
host_name = s3.amazonaws.com
initial_scan_datetime = default
interval = 1800
key_name = logs
max_items = 100000
max_retries = 10
recursion_depth = -1
sourcetype = aws:s3
whitelist = ^.
\/logs\/.*$

What am I missing ?

your help is much appreciated,
Matthias

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

you should probably configure a whitelist describing the files that you want: http://docs.splunk.com/Documentation/AddOns/latest/AWS/ConfigureInputs#S3_inputs

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...