Is there a way (using the Splunk TA for AWS or otherwise) that Splunk can connect to a publicly available S3 bucket (such as those made available here https://registry.opendata.aws/) and read in the data?
From the Splunk TA, the only buckets that I can read from are those which were created in my account.
After some further digging and testing it appears that it can be done but you need to create the input using the conf file as per link text
The key here for ingesting "old" data from a public dataset in S3 is that you need to set initial_scan_datetime
to be a date that is BEFORE the modified file time for the files in the S3 bucket.
Once I did this, I was able to pull the public dataset into Splunk from the public S3 bucket.
After some further digging and testing it appears that it can be done but you need to create the input using the conf file as per link text
The key here for ingesting "old" data from a public dataset in S3 is that you need to set initial_scan_datetime
to be a date that is BEFORE the modified file time for the files in the S3 bucket.
Once I did this, I was able to pull the public dataset into Splunk from the public S3 bucket.