Is my rex right? Rex has exceeded configured match_limit, consider raising the value in limits.conf

[samlinsongguo]

I am trying to extract about 20 fields from a log file each lines have about 800 charachers, I can only extract to first 14 field the get error saying my rex has exceeded configured match_limit, consider raising the value in limits.conf.

First, base on documentation the default value for match_limit is 100000. I am no where to close that limit,(I think)
Second I did try to check the configure file. but it doesnt seems working.
I created a limits.conf file in ....\etc\system\local\limits.conf. doesnt work
I changed ...\etc\system\default\limits.conf. doesnt work.

Any suggestion of where I am doing wrong? Is that possible is my rex(as below) not right?

^(.*?)"(?<SysProviderName>.+?(?="))(.*?){(?<SysProviderGuid>.+?(?=}))(.+?)<EventID>(?<EventID>.+?(?=<))(.+?)<EventName>(?<EventName>.+?(?=<))(.+?)<Version>(?<Version>.+?(?=<))(.+?)<Source>(?<Source>.+?(?=<))(.+?)<Level>(?<Level>.+?(?=<))(.+?)<Opcode>(?<Opcode>.+?(?=<))(.+?)<Keywords>(?<Keywords>.+?(?=<))(.+?)<Result>(?<Result>.+?(?=<))(.+?)"(?<Time>.+?(?="))(.+?)<Channel>(?<Channel>.+?(?=<))(.+?)<Computer>(?<Computer>.+?(?=<))(.+?)<ComputerUUID>(?<PCUUID>.+?(?=<))(.+?)SubjectUserName(?<SubjectUserName>.+?(?=<))(.+?)SubjectUserSid">(?<SubjectUserSid>.+?(?=<))(.+?)SubjectDomainName">(?<SubjectDomainName>.+?(?=<))(.+?)SubjectIP">(?<SubjectIP>.+?(?=<))(.+?)SubjectPort">(?<SubjectPort>.+?(?=<))

Thanks in advance.

[woodcock]

The answer is not to increase the limits; the answer is to make your RegEx more efficient. But we cannot help you do that because you have not given us the sample events and the field names/details.

[cpetterborg]

There is probably too much backtracking going on. Reduce the number of (?=<) that you have and perhaps the number of lazy matches, too. These constructs are great when you need them, but can create huge backtracking trees.

So where you are doing .+?(?=<) try instead to use [^<]+, which is essentially the same, but produces a much smaller tree.

[samlinsongguo]

HI Cpetterborg
Thank you for your tips what about "(.*?){" would it cause the same problem? should I replace it to [^{] as well?

[cpetterborg]

Replacing lookahead and lookbehind constructs will result in smaller match trees, so anytime you can eliminate them, your expressions will be more efficient. If you can provide an (obfuscated) example, I'd be happy to look at what will result in a much better regex. 🙂

[rey123]

@cpetterborg, I have an example, and would be grateful if you could kindly assist me with it:

Briefly, I have logs as follows:

ERROR [WorkerThreadMgmt] Pickup Job for execution failed.
com.domainname.schedulerservice.SchedulerServiceException: This module is using A3 Scheduler. But A3 Scheduler is not available on [http://a3random.domain.str.in.g:8080/rest/v1]. Please check if A3 Scheduler is enabled and [schedulerservice.url] is configured correctly in run.conf file. [Connection reset]

    at

org.apache.http.xyzl.clien.OpenableFtpClient.determineTarget(OpenableFtpClient.java:111)
at
org.apache.http.xyzl.clien.OpenableFtpClient.execute(OpenableFtpClient.java:111)
at
org.apache.http.xyzl.clien.OpenableFtpClient.execute(OpenableFtpClient.java:111)
at
org.apache.http.xyzl.clien.FlexibleFtpClient.execute(OpenableFtpClient.java:111)
at com.abcdomain.schedulerservice.SchedulerService$2.postReq(SchedulerService.java:123)
at com.abcdomain.schedulerservice.SchedulerService$2.postReq(SchedulerService.java:416)
at com.abcdomain.schedulerservice.SchedulerService$2.pickupJob(SchedulerService.java:509)
...

I need to extract the className, packageName and the Line number at which the exception is occurring:

| rex field=_raw "(?<packageName>com\\.abcdomain[\\.a-zA-Z\\d]+)\\.[\\w\\$]+\\.[\\w\\$]+\\((?<className>[\\w\\$]+)\\.java:(?<Line_number>\\d+)\\)"


| rex field=_raw "(?<packageName>com\\.abcdomain[\\.a-zA-Z\\d]+)\\.[\\w\\$]+\\.[\\w\\$]+\\((?<className>[\\w\\$]+)\\.java:(?<Line_number>\\d+)\\)"

However, while the regex works during testing, it fails with the

Streamed search execute failed because: Error in 'rex' command: Regex match error, please check log

error when encountering a large number of events, or when executed over, say, even 24 hours.

Would you have any idea how I could streamline the above regex to avoid such errors? Thanks! I tried to streamline the regex further but couldn't.