I am facing issue as per which S3 dashboard option in Splunk App for AWS is not loading any data. The Splunk documentation says that S3 data should be on-boarded using "sourcetype=aws:s3:accesslogs", however the query in S3 traffic tab has "aws:cloudtrai"l in it. I tried changing it to "aws:s3:accesslogs" but query is still not working out. I tried almost everything but it seems like a complicated issue as per which I will have to rebuilt the entire dashboard.
So, is it something that is a miss out from Splunk or myself?
First, make sure that you are sending your AWS data to the main index, if you created your own 'aws' index, then make sure to update the macros so that the summary indexes are properly populated. Here is a troubleshooting guide to help you.
Once you have set that up, then the queries should work and the source type 'aws:cloudtrail' will show the S3 access since the query is using the cloud trail API calls to S3 to show the access of the events.
That specific dashboard is using data models which are populated with the saved searches and macros mentioned in the troubleshooting guide.
First, make sure that you are sending your AWS data to the main index, if you created your own 'aws' index, then make sure to update the macros so that the summary indexes are properly populated. Here is a troubleshooting guide to help you.
Once you have set that up, then the queries should work and the source type 'aws:cloudtrail' will show the S3 access since the query is using the cloud trail API calls to S3 to show the access of the events.
That specific dashboard is using data models which are populated with the saved searches and macros mentioned in the troubleshooting guide.
I have tried most of the things to make that S3 dashboard work but it isn't.
Just to explain it little more, I am sending the data to main index with sourcetype="aws:s3:accesslogs" but it is still not working out. I am trying to work on some POC that will need this app as well, if possible then can we connect regarding this on call or email?
Query one the view in this dashboard is using.
(index="main" OR (index="main" OR index="default") OR index="default") sourcetype="aws:cloudtrail"
| lookup all_eventName eventName OUTPUTNEW function
| search function="S3 Data Event"
| spath output=bucketName path="requestParameters.bucketName"
| spath output=objectName path=requestParameters.key
| spath output=userName path=userIdentity.userName
| eval error=if(errorCode=="success",0, 1)
| lookup unauthorized_errorCode errorCode OUTPUT Unauthorized
| eval Unauthorized=if(Unauthorized=="true", 1, 0)
| stats count by region, aws_account_id, bucketName, objectName, userName, eventName, userAgent, sourceIPAddress,Unauthorized, error, readOnly,_time
| search (aws_account_id="") (region="")
| stats sum(count) as count sum(error) as errors
| nadefault errors count
| eval _total_name="Data Events", _total_field=count
There seems to be an issue with the field extraction for the userName. Try setting the 6th line to :
| spath output=userName path=userIdentity.sessionContext.sessionIssuer.userName
The search should complete with the count and errors tables populating. Here is the complete search, I took out the last 'search' since I'm not running it within the context of the search filters on the dashboard:
(index="main" OR (index="main" OR index="default") OR index="default") sourcetype="aws:cloudtrail"
| lookup all_eventName eventName OUTPUTNEW function
| search function="S3 Data Event"
| spath output=bucketName path="requestParameters.bucketName"
| spath output=objectName path=requestParameters.key
| spath output=userName path=userIdentity.sessionContext.sessionIssuer.userName
| eval error=if(errorCode=="success",0, 1)
| lookup unauthorized_errorCode errorCode OUTPUT Unauthorized
| eval Unauthorized=if(Unauthorized=="true", 1, 0)
| stats count by region, aws_account_id, bucketName, objectName, userName, userAgent, sourceIPAddress,Unauthorized, error, readOnly, _time
| stats sum(count) as count sum(error) as errors
| nadefault errors count
| eval _total_name="Data Events", _total_field=count
Hi Amiracle,
Thank you so much for providing that solution.
I followed your approach and was able to fix most of the issues but "Most Recent Modifications - Latest 10" is still not working out.
Below is the query:
(index="main" OR (index="main" OR index="default") OR index="default") sourcetype="aws:cloudtrail"
| lookup all_eventName eventName OUTPUTNEW function
| search function="S3 Data Event"
| spath output=bucketName path="requestParameters.bucketName"
| spath output=objectName path=requestParameters.key
| spath output=userName path=userIdentity.sessionContext.sessionIssuer.userName
| eval error=if(errorCode=="success",0, 1)
| lookup unauthorized_errorCode errorCode OUTPUT Unauthorized
| eval Unauthorized=if(Unauthorized=="true", 1, 0)
| stats count by region, aws_account_id, bucketName, objectName, userName, eventName, userAgent, sourceIPAddress,Unauthorized, error, readOnly,_time
| search (aws_account_id="") (region="")
| search readOnly=false (eventName="DeleteObject" OR eventName="PutObject" OR eventName="PostObject")
| sort - _time limit=10
| eval _time=strftime(_time, "%m/%d/%Y %H:%M:%S")
| table bucketName, objectName, userName, eventName, sourceIPAddress, userAgent, _time
The other tab "S3 - Traffic Analysis" is not working at all.
S3 Traffic Analysis is based on the data model: S3_Access_Log (Settings -> Data Models) and see if the data model was built. If not, hit the rebuild button or update it.
Yes, sure thanks a lot for help. Is there anything that you can suggest me to try before the model rebuilt ?
No, all the data is contained in the model so changing any of the searches would just lead to missing fields etc. Are you sending the data into Splunk using the SQS Based S3 input or are you using another method?
[aws_sqs_based_s3://S3_Logs]
aws_account = RoleName
aws_iam_role = RoleName
interval = 300
s3_file_decoder = S3AccessLogs
sourcetype = aws:s3:accesslogs
sqs_batch_size = 10
sqs_queue_region = us-east-1
sqs_queue_url = queueName
Any solution for "Most Recent Modifications - Latest 10" from S3 - Data events that is also not working out.
Yes, check the saved search "CloudTrail S3 Data Event Search" and see if it has completed with results. (Settings -> Search, reports and Alerts -> Find "CloudTrail S3 Data Event Search" and then click on Results.) If it's disabled, enable the search.
Yes, it is enabled and has completed the results.
That is the base search that is being used by this dashboard to help populate the different reports.