Splunk Enterprise Security

Azure: Is there a way Splunk, by default, extracts the fields from nested JSON logs?

yosplunksunny
New Member

Hi All,

Is there a way Splunk by default to extracts the fields from nested JSON logs? Right now Splunk is parsing the standard JSON files whoever it will not parse the value or ignore the nested JSON values however parse the rest of the data from the event.
Below is the sample event :
In this Splunk is parsing the data however field "policies":"["these values take as a single value rather parsing all other fields inside this field.

**{"time":"2019-09-23T10:52:06.6532250Z","resourceId":"/SUBSCRIPTIONS/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","operationName":"MICROSOFT.AUTHORIZATION/POLICIES/AUDIT/ACTION","category":"Action","resultType":"Success","resultSignature":"Succeeded.","durationMs":0,"callerIpAddress":"10.0.0.0","correlationId":"981cf31f-ffb2-415d-bbe4-j10d5637c393","identity":{"authorization":{"scope":"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","action":"Microsoft.Resources/checkPolicyCompliance/read","evidence":{"role":"Policy Insights Resource Provider Service Role","roleAssignmentScope":"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","roleAssignmentId":"30000e0edec8889988a018870ae4bdaf","roleDefinitionId":"30000e0edec8889688a018870ae4bdaf","principalId":"90111e0edec8889688a018870ae4bdaf","principalType":"ServicePrincipal"}},"claims":{"INR":"https://management.azure.com/","iss":"https://sts.windows.net/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/","iat":"1569235600","nbf":"5874","exp":"5656","aio":"10000+Y84Jsbb56565=","appid":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","appidacr":"2","http://schemas.microsoft.com/identity/claims/identityprovider":"https://sts.windows.net/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/","http://schemas.microsoft.com/identity/claims/objectidentifier":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","http://schemas.microsoft.com/identity/claims/tenantid":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","uti":"qbZDKv3YB0Six7cSv6eJFF","ver":"1.0"}},"level":"Warning","location":"global","properties":{"isComplianceCheck":"True","resourceLocation":null,"ancestors":"test-test-sandbox-non-production,test-non-production,test,111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","policies":"[{\"policyDefinitionId\":\"/providers/Microsoft.Authorization/policyDefinitions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/\",\"policySetDefinitionId\":\"/providers/Microsoft.Authorization/policySetDefinitions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/\",\"policyDefinitionReferenceId\":\"identityRemoveExternalAccountWithWritePermissionsMonitoring\",\"policySetDefinitionName\":\"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58\",\"policyDefinitionName\":\"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58\",\"policyDefinitionEffect\":\"AuditIfNotExists\",\"policyAssignmentId\":\"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/providers/Microsoft.Authorization/policyAssignments/SecurityCenterBuiltIn/\",\"policyAssignmentName\":\"SecurityCenterBuiltIn\",\"policyAssignmentScope\":\"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58\",\"policyAssignmentSku\":{\"name\":\"A1\",\"tier\":\"Standard\"},\"policyAssignmentParameters\":{}}]"},"am_category":"ACTION"}**

By default search time or index time , can we extract this nested values rather than using spath to extract in search.

As we need to use this is in the data model values once we get this field extracted.

Thanks in advance.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Have you looked at this answer from the archives? Essentially, the escaping is removed from the escaped double quotes, and the embedded field is unquoted, before parsing with spath, allowing the whole log to be parsed as you were hoping for.

0 Karma

snaikwade1
Observer

Hi,

I am more so looking for extracting these fields and them making them CIM compliant. The microsoft cloud services add-on does not help much as the fields are not extracted and data model mapping is not done correctly. 

0 Karma

to4kawa
Ultra Champion

props.conf
[your sourcetype]
INDEXED_EXTRACTIONS = json
TRANSFORMS-azure_sample

transforms.conf
[azure_sample]
SOURCE_KEY=properties.policies
REGEX = \"(?<name>[^\"]+)\":\"(?<value>[^\"]+)\"
FORMAT = $1::"$2"
REPEAT_MATCH = true
WRITE_META = true

I haven't tried it, but how about this?

0 Karma

snaikwade1
Observer

Hi,

were you able to resolve the issue ?

0 Karma

eltra1n
New Member

Looking in to this as well, will post results. Do we know why  events are treated differently?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...