Not all logs are being imported from S3?

[Reosoul]

I have AWS log data stored in an S3 bucket and have linked to it in Splunk through "Splunk App for AWS." I see the first two month's of data files (there is a large number of very small files that get generated), but nothing after.

Is there a setting or something else that may be limiting this?

[jzhong_splunk]

Hi Reosoul, the Splunk App for AWS uses the AWS TA to collect the data. For such advanced settings, it's not exposed on the App UI (as we try to make the app configuration as easy as possible). You can open the 'Splunk Add-on for AWS', filter by Service:S3, and edit the modular input. In the 3rd tab 'Template', yo can set the "Start Date/Time" value. Sorry it's a plain text filed. The format of the value can be found on the TA documentation: http://docs.splunk.com/Documentation/AddOns/latest/AWS/S3

initial_scan_datetime
Relevant only the first time the input runs, this value indicates the oldest file time that should match a file scan, using ISO8601 formatting (for example, 2011-07-06T21:54:23.000-07:00). This value defaults to seven days ago, calculated back from the date you configure the input stanza.

[Reosoul]

Is that not just setting a start point for when the data starts collecting?

I am trying to be able to collect and analyze data up to maybe a year.