Splunk Search

How to line break events

anasamer
New Member

Can anyone here help with breaking this sample into multiple events each should start with { "resourceId": ?
I have the below log sample:

{"records": [{ "resourceId": "/SUBSCRIPTIONS/9799XXX5-F9BF-XXXX-XXXX-6DDXXXXXF4D99/RESOURCEGROUPS/DEVELOPMENT/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/SPLUNKWAF", "operationName": "ApplicationGatewayFirewall", "time": "2019-06-11T11:09:39.2282087Z", "category": "ApplicationGatewayFirewallLog", "properties": {
  "instanceId": "ApplicationGatewayRole_IN_1",
  "clientIp": "10.0.1.5",
  "clientPort": "0",
  "requestUri": "/en-GB/splunkd/__raw/services/server/health/splunkd?output_mode=json&_=1560250716771",
  "ruleSetType": "OWASP",
  "ruleSetVersion": "3.0",
  "ruleId": "920350",
  "ruleGroup": "920-PROTOCOL-ENFORCEMENT",
  "message": "Host header is a numeric IP address",
  "action": "Matched",
  "site": "Global",
  "details": {
    "message": "Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host.",
    "data": "10.10.10.10:8001",
    "file": "rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf",
    "line": "791"
  },
  "hostname": "10.10.10.10",
  "transactionId": "17726168135477758612"
}},{ "resourceId": "/SUBSCRIPTIONS/9799XXX5-F9BF-XXXX-XXXX-6DDXXXXXF4D99/RESOURCEGROUPS/DEVELOPMENT/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/SPLUNKWAF", "operationName": "ApplicationGatewayFirewall", "time": "2019-06-11T11:09:43.2069335Z", "category": "ApplicationGatewayFirewallLog", "properties": {
  "instanceId": "ApplicationGatewayRole_IN_1",
  "clientIp": "10.0.1.5",
  "clientPort": "0",
  "requestUri": "/en-GB/splunkd/__raw/services/messages?output_mode=json&sort_key=timeCreated_epochSecs&sort_dir=desc&count=1000&_=1560250720227",
  "ruleSetType": "OWASP",
  "ruleSetVersion": "3.0",
  "ruleId": "920350",
  "ruleGroup": "920-PROTOCOL-ENFORCEMENT",
  "message": "Host header is a numeric IP address",
  "action": "Matched",
  "site": "Global",
  "details": {
    "message": "Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host.",
    "data": "10.10.10.10:8001",
    "file": "rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf",
    "line": "791"
  },
  "hostname": "10.10.10.10",
  "transactionId": "17726168135477758613"
}},{ "resourceId": "/SUBSCRIPTIONS/9799XXX5-F9BF-XXXX-XXXX-6DDXXXXXF4D99/RESOURCEGROUPS/DEVELOPMENT/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/SPLUNKWAF", "operationName": "ApplicationGatewayFirewall", "time": "2019-06-11T11:09:49.9545793Z", "category": "ApplicationGatewayFirewallLog", "properties": {
  "instanceId": "ApplicationGatewayRole_IN_1",
  "clientIp": "10.0.1.5",
  "clientPort": "0",
  "requestUri": "/en-GB/splunkd/__raw/services/messages?output_mode=json&sort_key=timeCreated_epochSecs&sort_dir=desc&count=1000&_=1560250716774",
  "ruleSetType": "OWASP",
  "ruleSetVersion": "3.0",
  "ruleId": "920350",
  "ruleGroup": "920-PROTOCOL-ENFORCEMENT",
  "message": "Host header is a numeric IP address",
  "action": "Matched",
  "site": "Global",
  "details": {
    "message": "Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host.",
    "data": "10.10.10.10:8001",
    "file": "rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf",
    "line": "791"
  },
  "hostname": "10.10.10.10",
  "transactionId": "17726168135477758614"
}}]}
{"records": [{ "resourceId": "/SUBSCRIPTIONS/9799XXX5-F9BF-XXXX-XXXX-6DDXXXXXF4D99/RESOURCEGROUPS/DEVELOPMENT/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/SPLUNKWAF", "operationName": "ApplicationGatewayFirewall", "time": "2019-06-11T11:08:59.1006429Z", "category": "ApplicationGatewayFirewallLog", "properties": {
  "instanceId": "ApplicationGatewayRole_IN_0",
  "clientIp": "10.0.1.7",
  "clientPort": "0",
  "requestUri": "/en-GB/splunkd/__raw/services/server/health/splunkd?output_mode=json&_=1560250716762",
  "ruleSetType": "OWASP",
  "ruleSetVersion": "3.0",
  "ruleId": "920350",
  "ruleGroup": "920-PROTOCOL-ENFORCEMENT",
  "message": "Host header is a numeric IP address",
  "action": "Matched",
  "site": "Global",
  "details": {
    "message": "Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host.",
    "data": "10.10.10.10:8001",
    "file": "rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf",
    "line": "791"
  },
  "hostname": "10.10.10.10",
  "transactionId": "17365880165288120552"
}}]}
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you tried LINE_BREAKER = ()\{ "resourceId": ?

---
If this reply helps you, Karma would be appreciated.
0 Karma

anasamer
New Member

nope it is not working

alt text

0 Karma

FrankVl
Ultra Champion

Your image is not publicly visible. Make sure to fix the typo (you need capital I instead of lowercase).

0 Karma

FrankVl
Ultra Champion

The i in resourceId must be a capital I 🙂

Of course to be combined with SHOULD_LINEMERGE = false.

And a bit more specific linebreak to try could be: LINE_BREAKER = ((?:\]\})?[\r\n\s]*\{"records":\s\[|,)\{\s"resourceId":
That also strips out the , in between events and the records [ bit.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I fixed the 'I'.

---
If this reply helps you, Karma would be appreciated.
0 Karma

anasamer
New Member

thanks @FrankVl this regex captured the records in the middle of the log but not capturing the first records

0 Karma

anasamer
New Member

regexr.com/4flnp

0 Karma

FrankVl
Ultra Champion

That is not the same regex I shared 😉

You added a closing } in between the character class that is matching linebreaks and the * behind it. And then indeed it doesn't work properly. Your fix is also incorrect, as it should be a * to add, not a ?, since there could be a combination of multiple linebreak and whitespace characters.

But you simply shouldn't add that } there in the first place, as it will result in stripping that off from the end of the previous event and that will break your json syntax.

I think it should work if you use the exact regex I shared, but do let me know if it doesn't: https://regexr.com/4flu5

0 Karma

anasamer
New Member

I fixed it by adding ? so it will be like

((?:\]\})*[\r\n\s]?}*\{"records":\s\[|,)\{\s*"resourceId":

Thanks

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...