All Apps and Add-ons

Splunk Add-On for S3 data inputs question

pkeller
Contributor

In our test environment, we successfully setup the Splunk Add-on for Amazon S3 and pulled buckets so that we could view the data and make sure the props.conf settings were sorted out before we moved to production.

When we setup the same configuration in Production, we're only pulling 'new' buckets. We'd like to ingest all the same buckets that were pulled into our test environment. Is there some setting in the Add-on (or on the S3 side) that keeps track of what has already been pulled, thus preventing a duplicate pull?

Thanks very much,

Tags (2)
1 Solution

ryandg
Communicator

When you say you moved it from test to production, do they share the same devices that pull from S3 and just different indexers? The only thing I can think of is the pointers were already created so when you switched the outputs to a different indexer cluster you kept the previous pointers. Easiest way is to just clone/recreate the S3 inputs (assuming there aren't a ton of them) and it will reload them.

View solution in original post

0 Karma

pkeller
Contributor

Ultimately I just wiped and reinstalled the app and reconfigured the inputs. The form has a place to enter the date that you're choosing to go back to, but after the first collection, the app seems to look somewhere else last_modified=2016-04-07T2 instead of the date that you enter via the UI ... gets it from

index_store.last_modified in s3_mod/aws_s3_data_loader.py

Anyway. Thank you. I'm all caught up now.

0 Karma

ryandg
Communicator

When you say you moved it from test to production, do they share the same devices that pull from S3 and just different indexers? The only thing I can think of is the pointers were already created so when you switched the outputs to a different indexer cluster you kept the previous pointers. Easiest way is to just clone/recreate the S3 inputs (assuming there aren't a ton of them) and it will reload them.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...