Splunk Search

How to use line breaking regex for Shibboleth audit logs?

jwalzerpitt
Influencer

We are using Kafka Connect and we just started to ingest Shib audit logs. I am getting a bunch of events all rolled into one so I tried the splunk.hec.raw.line.breaker":"####", line in our JSON file on the Kafka Connect server as this matches the line I use on other log sources such as Cisco, Palo Alto, etc., and breaks the line just fine, but in this case, it's not working.

The events look like this:

{"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|username@foo.com|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"}{"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|username@foo.com|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"}{"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|username@foo.com|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"}

What would be the regex to use to line break?

I tried using ,"EventTime":"(?>\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})" but the events aren't breaking quite right as I see:

-31 15:25:51"}{"EventReceivedTime":"2019-05-31

at the start of each event

Thx

0 Karma
1 Solution

aromanauskas
Path Finder

This should get you what you're looking for. Since there is Nothing between your events you need an empty capture group. This will break if there are any closing braces in the events though.

In props.conf
LINE_BREAKER = }()
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TIME_PREFIX = {"EventReceivedTime":"

If you are 100% sure that EventTime will always be at the end.
LINE_BREAKER = "EventTime":"\d+-\d+-\d+\s+\d+:\d+:\d+"}()

View solution in original post

0 Karma

aromanauskas
Path Finder

This should get you what you're looking for. Since there is Nothing between your events you need an empty capture group. This will break if there are any closing braces in the events though.

In props.conf
LINE_BREAKER = }()
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TIME_PREFIX = {"EventReceivedTime":"

If you are 100% sure that EventTime will always be at the end.
LINE_BREAKER = "EventTime":"\d+-\d+-\d+\s+\d+:\d+:\d+"}()

0 Karma

jwalzerpitt
Influencer

Worked perfectly - thx a million!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...