All Apps and Add-ons

How to connect s3 buckets in Glacier to the Splunk App for AWS?

cwyse
Explorer

We are having problems with s3 bucket injection. Our corporate security policy states we need to keep 2 years of our ELb logs in s3. So we lifecycle them into glacier. This unfortunately means that when we try to connect to them with the AWS app, we get a whole lot of files with a different storage type and this causes thousands of errors. For some reason, this kills the process so we can't get new data. When I look at the source, I do see the number going up, but we can't seem to get anything newer than the first time this ran which was about a week ago.

These are the sanitized errors we see constantly scrolling through aws_s3.log:

2015-10-20 22:55:12,240 ERROR pid=22473 tid=MainThread file=aws_s3.py:stream_events:868 | Incomplete: bucket: 'OURBUCKET' key: u'OURPREFIX/AWSLogs/OURACCOUNT#/elasticloadbalancing/OURZONE/2014/05/19/OURACCOUNT#_elasticloadbalancing_OURZONE_OURELBNAME_20140519T0100Z_10.241.4.64_3indsmej.log' etag: "2464d51635ffc8954e36b59f479f72cc" attempt_number: 2 orig_size: 1119509 bytes_streamed: 0 total_bytes_streamed: 0 Exception: S3ResponseError: 403 Forbidden - InvalidObjectState - The operation is not valid for the object's storage class
0 Karma

azhang_splunk
Splunk Employee
Splunk Employee

Is there any data in s3 bucket "OURBUCKET"? I think that's just empty bucket for listing purpose only. If so, you can just skip that folder either use whitelist or folder name filter in config page.

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...