About danint

danint · ‎01-18-2022

Haven't looked into this too much, but I'm guessing it is related to the updates in version 5.2.1: https://docs.splunk.com/Documentation/AddOns/released/AWS/Releasenotes """ Version 5.2.1 of the Splunk Add-on for AWS version contains the following new and changed features: Added validation of the signature of the SNS message being sent from SQS queue to Splunk. The source of the logs is validated by matching the signature of the SNS message with the signature field. """ The setup docs for the CloudTrail input state S3->SNS->SQS now. I don't think this was the case before, but I don't have a copy of the old docs to check. It looks like it is using this plugin to validate. https://pypi.org/project/validate-aws-sns-message/ Note, it "Requires message be no older than one hour, the maximum lifetime of an SNS message." Splunk, it would be much better if this was an optional feature, as it breaks our infrastructure, especially bad since this is a PATCH update, not MAJOR.

danint · ‎11-21-2019

Hello, There are two bugs in the Splunk Add-on for Amazon Web Services regarding S3 Access Logs (using Generic S3 Input) 1) Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_common.py line 27 asc.aws_s3_accesslogs: r".+\d{4}-\d{2}-\d{2}-\d{2}-\d{2}-\d{2}-.+$" Per Amazon S3 Server Access Logging, the log file key format is: TargetPrefixYYYY-mm-DD-HH-MM-SS-UniqueString , where TargetPrefix is an "(Optional) A prefix for Amazon S3 to assign to all log object keys.". The regex on line 27 makes TargetPrefix required and fails to fetch logs without a prefix. It should instead be: asc.aws_s3_accesslogs: r".*\d{4}-\d{2}-\d{2}-\d{2}-\d{2}-\d{2}-.+$" 2) Splunk_TA_aws/default/props.conf line 441 EXTRACT-s3 = ^\s*(?P<bucket_owner>[^\s]+)\s+(?P<bucket_name>[^\s]+)\s+\[(?P<request_time>.+)\]\s+(?P<remote_ip>[^\s]+)\s+(?P<requester>[^\s]+)\s+(?P<request_id>[^\s]+)\s+(?P<operation>[^\s]+)\s+(?P<key>[^\s]+)\s+"(?P<request_uri>.+)"\s+(?P<http_status>[^\s]+)\s+(?P<error_code>[^\s]+)\s+(?P<bytes_sent>[^\s]+)\s+(?P<object_size>[^\s]+)\s+(?P<total_time>[^\s]+)\s+(?P<turn_around_time>[^\s]+)\s+"(?P<referrer>.+)"\s+"(?P<user_agent>.+)"\s+(?P<version_id>[-\w]+)$ Per Amazon S3 Server Access Log Format, AWS "might extend the access log record format by adding new fields to the end of each line. Code that parses server access logs must be written to handle trailing fields that it does not understand." The regex on line 441 does not allow trailing fields. it should instead be: EXTRACT-s3 = ^\s*(?P<bucket_owner>[^\s]+)\s+(?P<bucket_name>[^\s]+)\s+\[(?P<request_time>.+)\]\s+(?P<remote_ip>[^\s]+)\s+(?P<requester>[^\s]+)\s+(?P<request_id>[^\s]+)\s+(?P<operation>[^\s]+)\s+(?P<key>[^\s]+)\s+"(?P<request_uri>.+)"\s+(?P<http_status>[^\s]+)\s+(?P<error_code>[^\s]+)\s+(?P<bytes_sent>[^\s]+)\s+(?P<object_size>[^\s]+)\s+(?P<total_time>[^\s]+)\s+(?P<turn_around_time>[^\s]+)\s+"(?P<referrer>.+)"\s+"(?P<user_agent>.+)"\s+(?P<version_id>[-\w]+) ``Or, to preserve the trailing fields, it could be extended to parse them into a "other_fields" field or something similar. I can't post links, so search for the bolded titles to find the relevant AWS docs.

Posts	2
Solutions	0
Karma Given	0
Karma Received	4
Member Since	‎11-21-2019

Online Status	Offline
Date Last Visited	‎01-18-2022 05:52 PM

Splunk AWS TA S3 Access Logs Bugs

Re: AWS CloudTrail SQS Based S3 error

Splunk AWS TA S3 Access Logs Bugs