Can anyone here help with breaking this sample into multiple events each should start with { "resourceId": ?
I have the below log sample:
{"records": [{ "resourceId": "/SUBSCRIPTIONS/9799XXX5-F9BF-XXXX-XXXX-6DDXXXXXF4D99/RESOURCEGROUPS/DEVELOPMENT/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/SPLUNKWAF", "operationName": "ApplicationGatewayFirewall", "time": "2019-06-11T11:09:39.2282087Z", "category": "ApplicationGatewayFirewallLog", "properties": {
"instanceId": "ApplicationGatewayRole_IN_1",
"clientIp": "10.0.1.5",
"clientPort": "0",
"requestUri": "/en-GB/splunkd/__raw/services/server/health/splunkd?output_mode=json&_=1560250716771",
"ruleSetType": "OWASP",
"ruleSetVersion": "3.0",
"ruleId": "920350",
"ruleGroup": "920-PROTOCOL-ENFORCEMENT",
"message": "Host header is a numeric IP address",
"action": "Matched",
"site": "Global",
"details": {
"message": "Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host.",
"data": "10.10.10.10:8001",
"file": "rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf",
"line": "791"
},
"hostname": "10.10.10.10",
"transactionId": "17726168135477758612"
}},{ "resourceId": "/SUBSCRIPTIONS/9799XXX5-F9BF-XXXX-XXXX-6DDXXXXXF4D99/RESOURCEGROUPS/DEVELOPMENT/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/SPLUNKWAF", "operationName": "ApplicationGatewayFirewall", "time": "2019-06-11T11:09:43.2069335Z", "category": "ApplicationGatewayFirewallLog", "properties": {
"instanceId": "ApplicationGatewayRole_IN_1",
"clientIp": "10.0.1.5",
"clientPort": "0",
"requestUri": "/en-GB/splunkd/__raw/services/messages?output_mode=json&sort_key=timeCreated_epochSecs&sort_dir=desc&count=1000&_=1560250720227",
"ruleSetType": "OWASP",
"ruleSetVersion": "3.0",
"ruleId": "920350",
"ruleGroup": "920-PROTOCOL-ENFORCEMENT",
"message": "Host header is a numeric IP address",
"action": "Matched",
"site": "Global",
"details": {
"message": "Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host.",
"data": "10.10.10.10:8001",
"file": "rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf",
"line": "791"
},
"hostname": "10.10.10.10",
"transactionId": "17726168135477758613"
}},{ "resourceId": "/SUBSCRIPTIONS/9799XXX5-F9BF-XXXX-XXXX-6DDXXXXXF4D99/RESOURCEGROUPS/DEVELOPMENT/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/SPLUNKWAF", "operationName": "ApplicationGatewayFirewall", "time": "2019-06-11T11:09:49.9545793Z", "category": "ApplicationGatewayFirewallLog", "properties": {
"instanceId": "ApplicationGatewayRole_IN_1",
"clientIp": "10.0.1.5",
"clientPort": "0",
"requestUri": "/en-GB/splunkd/__raw/services/messages?output_mode=json&sort_key=timeCreated_epochSecs&sort_dir=desc&count=1000&_=1560250716774",
"ruleSetType": "OWASP",
"ruleSetVersion": "3.0",
"ruleId": "920350",
"ruleGroup": "920-PROTOCOL-ENFORCEMENT",
"message": "Host header is a numeric IP address",
"action": "Matched",
"site": "Global",
"details": {
"message": "Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host.",
"data": "10.10.10.10:8001",
"file": "rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf",
"line": "791"
},
"hostname": "10.10.10.10",
"transactionId": "17726168135477758614"
}}]}
{"records": [{ "resourceId": "/SUBSCRIPTIONS/9799XXX5-F9BF-XXXX-XXXX-6DDXXXXXF4D99/RESOURCEGROUPS/DEVELOPMENT/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/SPLUNKWAF", "operationName": "ApplicationGatewayFirewall", "time": "2019-06-11T11:08:59.1006429Z", "category": "ApplicationGatewayFirewallLog", "properties": {
"instanceId": "ApplicationGatewayRole_IN_0",
"clientIp": "10.0.1.7",
"clientPort": "0",
"requestUri": "/en-GB/splunkd/__raw/services/server/health/splunkd?output_mode=json&_=1560250716762",
"ruleSetType": "OWASP",
"ruleSetVersion": "3.0",
"ruleId": "920350",
"ruleGroup": "920-PROTOCOL-ENFORCEMENT",
"message": "Host header is a numeric IP address",
"action": "Matched",
"site": "Global",
"details": {
"message": "Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host.",
"data": "10.10.10.10:8001",
"file": "rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf",
"line": "791"
},
"hostname": "10.10.10.10",
"transactionId": "17365880165288120552"
}}]}
Have you tried LINE_BREAKER = ()\{ "resourceId":
?
nope it is not working
Your image is not publicly visible. Make sure to fix the typo (you need capital I instead of lowercase).
The i in resourceId must be a capital I 🙂
Of course to be combined with SHOULD_LINEMERGE = false
.
And a bit more specific linebreak to try could be: LINE_BREAKER = ((?:\]\})?[\r\n\s]*\{"records":\s\[|,)\{\s"resourceId":
That also strips out the , in between events and the records [
bit.
I fixed the 'I'.
thanks @FrankVl this regex captured the records in the middle of the log but not capturing the first records
regexr.com/4flnp
That is not the same regex I shared 😉
You added a closing }
in between the character class that is matching linebreaks and the *
behind it. And then indeed it doesn't work properly. Your fix is also incorrect, as it should be a *
to add, not a ?
, since there could be a combination of multiple linebreak and whitespace characters.
But you simply shouldn't add that }
there in the first place, as it will result in stripping that off from the end of the previous event and that will break your json syntax.
I think it should work if you use the exact regex I shared, but do let me know if it doesn't: https://regexr.com/4flu5
I fixed it by adding ? so it will be like
((?:\]\})*[\r\n\s]?}*\{"records":\s\[|,)\{\s*"resourceId":
Thanks