All Apps and Add-ons

Splunk Add-On for S3 data inputs question

pkeller
Contributor

In our test environment, we successfully setup the Splunk Add-on for Amazon S3 and pulled buckets so that we could view the data and make sure the props.conf settings were sorted out before we moved to production.

When we setup the same configuration in Production, we're only pulling 'new' buckets. We'd like to ingest all the same buckets that were pulled into our test environment. Is there some setting in the Add-on (or on the S3 side) that keeps track of what has already been pulled, thus preventing a duplicate pull?

Thanks very much,

Tags (2)
1 Solution

ryandg
Communicator

When you say you moved it from test to production, do they share the same devices that pull from S3 and just different indexers? The only thing I can think of is the pointers were already created so when you switched the outputs to a different indexer cluster you kept the previous pointers. Easiest way is to just clone/recreate the S3 inputs (assuming there aren't a ton of them) and it will reload them.

View solution in original post

0 Karma

pkeller
Contributor

Ultimately I just wiped and reinstalled the app and reconfigured the inputs. The form has a place to enter the date that you're choosing to go back to, but after the first collection, the app seems to look somewhere else last_modified=2016-04-07T2 instead of the date that you enter via the UI ... gets it from

index_store.last_modified in s3_mod/aws_s3_data_loader.py

Anyway. Thank you. I'm all caught up now.

0 Karma

ryandg
Communicator

When you say you moved it from test to production, do they share the same devices that pull from S3 and just different indexers? The only thing I can think of is the pointers were already created so when you switched the outputs to a different indexer cluster you kept the previous pointers. Easiest way is to just clone/recreate the S3 inputs (assuming there aren't a ton of them) and it will reload them.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...