Getting Data In

Json field parsing

martinnepolean
Explorer

Hi,

We are getting the aws macie events as _json souretype, due to multiple loops there is a problem in fields extraction. I have give the screenshots below, red oval should be the field name and green oval should be valued.

for example the field name is detail.summary events.createtags.isp amazon and value is 436, but we need to field name till ISP and value to be amazon.

Please let me know how to get it gone through props and transforms

alt textalt text

0 Karma

to4kawa
Ultra Champion
| makeresults 
| eval _raw="{\"version\":\"0\",\"id\":\"2561455-c673-0hy6-673b-447895415\",\"detail-type\":\"Macie Alert\",\"source\":\"aws.macie\",\"account\":\"123456789\",\"time\":\"2020-01-07T10:46:36Z\",\"region\":\"us-east-1\",\"resources\":[\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb\",\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7\"],\"detail\":{\"notification-type\":\"ALERT_UPDATED\",\"tags\":[\"Suspicious Access\",\"Basic Alert\"],\"name\":\"Access Denied to IAM user while attempting to get an AWS S3 Object from outside of AWS\",\"severity\":\"LOW\",\"url\":\"https://mt.us-east-1.macie.aws.amazon.com/posts/arn%3Aaws%3Amacie%3Aus-east-1%3A123456789%3Atrigger%2F0c54ddb4cd37e6b8316ecdc1ba4ae3b7%2Falert%2F014f2161de2fffc59dd5d2cdf81a73fb\",\"alert-arn\":\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb\",\"risk-score\":3,\"updated-at\":\"2020-01-07T10:46:36.136911\",\"created-at\":\"2020-01-07T00:46:35.139000+00:00\",\"actor\":\"321404829113:anonymous_principal\",\"summary\":{\"Description\":\"Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content\",\"IP\":{\"216.20.176.6\":1,\"216.20.176.5\":1,\"216.20.176.4\":2,\"216.20.176.2\":3},\"Time Range\":[{\"count\":3,\"start\":\"2020-01-07T00:29:44Z\",\"end\":\"2020-01-07T00:29:47Z\"},{\"count\":1,\"start\":\"2020-01-07T10:06:11Z\",\"end\":\"2020-01-07T10:06:11Z\"},{\"count\":1,\"start\":\"2020-01-07T07:51:59Z\",\"end\":\"2020-01-07T07:51:59Z\"},{\"count\":1,\"start\":\"2020-01-07T10:19:18Z\",\"end\":\"2020-01-07T10:19:18Z\"},{\"count\":1,\"start\":\"2020-01-07T10:24:37Z\",\"end\":\"2020-01-07T10:24:37Z\"}],\"Record Count\":5,\"Location\":{\"us-east-1\":7},\"Event Count\":7,\"Events\":{\"GetObject\":{\"count\":5,\"ISP\":{\"Company\":5},\"Error Code\":{\"AccessDenied\":5}},\"ListObjects\":{\"count\":2,\"ISP\":{\"Company\":2},\"Error Code\":{\"AccessDenied\":2}}},\"recipientAccountId\":{\"321404829113\":7}},\"trigger\":{\"rule-arn\":\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7\",\"alert-type\":\"basic\",\"created-at\":\"2019-12-19 09:32:37.931000+00:00\",\"description\":\"Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content\",\"risk\":3}}}"
| eval ErrorCode=spath(_raw,"detail.summary.Events.GetObject.Error Code")
| eval IP=spath(_raw,"detail.summary.IP")
| rex field=ErrorCode "(?<ErrorCode>\w+)"
| rex field=IP max_match=20 "(?<IP>(?<=\")[\w.]+)"
| fields - _*

Hi, @martinnepolean
The fields for the first question are missing, is this OK?

0 Karma

martinnepolean
Explorer

I am looking to get all fields not only errorcode and IP. any help?

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@martinnepolean

Can you please share your expected output from the sample you have shared?

0 Karma

martinnepolean
Explorer

Hi Kamlesh,

One of the example to show how the field extraction happens, currently we are getting like below one

Detail.Summary.Events.GetObject.Error Code.AccessDenied=2

But we like to extract it as
Detail.Summary.Events.GetObject.Error Code = AccessDenied

and Below is the list of fields I am looking for from the above event shared.

version=0
id=2561455-c673-0hy6-673b-447895415
detail-type=Maciealert
source=aws.macie
account=123456789
time=2020-01-07T10:46:36Z
region=us-east-1
resources=arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb","arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7"
detail.notification-type=ALERT_UPDATED
tags=Suspicious Access,Basic Alert
name=Access Denied to IAM user while attempting to get an AWS S3 Object from outside of AWS"
severity=LOW
url=https://mt.us-east-1.macie.aws.amazon.com/posts/arn%3Aaws%3Amacie%3Aus-east-1%3A123456789%3Atrigger%...
alert-arn=arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb
risk-score=3
updated-at=2020-01-07T10:46:36.136911
created-at=2020-01-07T00:46:35.139000+00:00
actor=321404829113:anonymous_principal
summary.Description=Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content
IP=216.20.176.6,216.20.176.5,216.20.176.4,216.20.176.2
TimeRange="start":"2020-01-07T00:29:44Z","end":"2020-01-07T00:29:47Z","start":"2020-01-07T10:06:11Z","end":"2020-01-07T10:06:11Z","start":"2020-01-07T07:51:59Z","end":"2020-01-07T07:51:59Z","start":"2020-01-07T10:19:18Z","end":"2020-01-07T10:19:18Z","start":"2020-01-07T10:24:37Z","end":"2020-01-07T10:24:37Z"
Location="us-east-1
Event Count=7
Events.GetObject.count=5
Events.GetObject.ISP=Company
Events.GetObject.Error Code=AccessDenied
Events.ListObjects.count=2
Events.ListObjects.ISP=Company
Events.ListObjects.Error Code=AccessDenied
recipientAccountId=321404829113
trigger.rule-arn=arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7"
trigger.alert-type=basic
trigger.created-at=2019-12-19 09:32:37.931000+00:00
trigger.description=Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content
trigger.risk=3

0 Karma

to4kawa
Ultra Champion

spath, props.conf 's JSON extract is like this.
It is inevitable.

0 Karma

to4kawa
Ultra Champion

your sample doesn't have detail.summary events.createtags.isp
you say all fields. which?
Apparently the fields have been extracted.

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@martinnepolean
Can you please share sample _raw events and expected output?

0 Karma

martinnepolean
Explorer

{"version":"0","id":"2561455-c673-0hy6-673b-447895415","detail-type":"Macie Alert","source":"aws.macie","account":"123456789","time":"2020-01-07T10:46:36Z","region":"us-east-1","resources":["arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb","arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7"],"detail":{"notification-type":"ALERT_UPDATED","tags":["Suspicious Access","Basic Alert"],"name":"Access Denied to IAM user while attempting to get an AWS S3 Object from outside of AWS","severity":"LOW","url":"https://mt.us-east-1.macie.aws.amazon.com/posts/arn%3Aaws%3Amacie%3Aus-east-1%3A123456789%3Atrigger%... Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content","IP":{"216.20.176.6":1,"216.20.176.5":1,"216.20.176.4":2,"216.20.176.2":3},"Time Range":[{"count":3,"start":"2020-01-07T00:29:44Z","end":"2020-01-07T00:29:47Z"},{"count":1,"start":"2020-01-07T10:06:11Z","end":"2020-01-07T10:06:11Z"},{"count":1,"start":"2020-01-07T07:51:59Z","end":"2020-01-07T07:51:59Z"},{"count":1,"start":"2020-01-07T10:19:18Z","end":"2020-01-07T10:19:18Z"},{"count":1,"start":"2020-01-07T10:24:37Z","end":"2020-01-07T10:24:37Z"}],"Record Count":5,"Location":{"us-east-1":7},"Event Count":7,"Events":{"GetObject":{"count":5,"ISP":{"Company":5},"Error Code":{"AccessDenied":5}},"ListObjects":{"count":2,"ISP":{"Company":2},"Error Code":{"AccessDenied":2}}},"recipientAccountId":{"321404829113":7}},"trigger":{"rule-arn":"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7","alert-type":"basic","created-at":"2019-12-19 09:32:37.931000+00:00","description":"Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content","risk":3}}}

I like to have error code as an field and AccessDenied as value which is not showing up now. and like to tag all IP addresses under IP field name.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...