How do I make a subsearch run faster...or is the search itself incorrect?

[MasterOogway]

When I run the following subsearch over an hours time it takes many minutes, if it completes at all. When run over Realtime it is basically useless. What the customer is attempting to do is search for something with -[0-4]- (regex takes care of this) in it and when found check to see if it matches the string, %SYS-3-PORT_DOT3_ALIGNMENT_ERROR

sourcetype=syslog_info %SYS-3-PORT_DOT3_ALIGNMENT_ERROR [search sourcetype=syslog_info | rex _raw=”\%\w+\-\w+\-\b([0-4])\-\w+” ]

I am looking for any reasons/ideas why this search is so intense that it fails. The kicker is I need to do this on another 900+ searches so it imperative I figure out what the bottleneck is. Thoughts?

Master Oogway

[ftk]

What is the reason for doing the subsearch with rex if in the end you are just looking for a single value within the rex matches? The example seems very inefficient to me. Since you are looking for a single value only and are performing the subsearch on the same index and sourcetype as the outer search, you might as well search for sourcetype=syslog_info %SYS-3-PORT_DOT3_ALIGNMENT_ERROR only and be done, cut the whole subsearch out of the picture.

[Edit: Response to comments]
First I would start with a field extraction to make comparisons against your lookup table easier. Something like this in props.conf:

[syslog_info]
EXTRACT-my_field = (?<error>\%\w+\-\w+\-\b([0-4])\-\w+)

Then I would create a lookup csv with two columns and populate it with your ~900 expected events:

message, expected
SYS.*-2-MOD_RESET, true
C4K_IOSMODPORTMAN-4-POWERSUPPLYGOOD, true
SECURITY.*-1-PORTSHUTDOWN, true
SYS-3-PORT_DOT3_ALIGNMENT_ERROR, true

and place it in the lookups directory.

Now I would do a search similar to the following to display all events that I did not expect:

sourcetype=syslog_info | lookup my_lookup.csv my_field AS message OUTPUT expected | where expected <> "true"

[Edit 2: response to new comments]
To do a lookup against a three column csv (message, device, expected) you would use a lookup statement like this:

sourcetype=syslog_info | lookup my_lookup.csv my_field AS message host AS device OUTPUT expected | where expected <> "true"

Now the logic of emailing different groups based on a false or a no match is a little harder to achieve. To do this I would actually set up two saved searches, each with different alert actions.

Both searches would be like this:

sourcetype=syslog_info | lookup my_lookup.csv my_field AS message host AS device OUTPUT expected

and then I would set up different alert conditions. For search 1 I would use a custom condition of where expected = "true" and as an alert action email group A. For search 2 I would use a custom condition of where expected <> "true" and as an alert action have it email group B.

I hope I understood your requirements correctly.

[ftk]

If you are defining the extraction in props.conf, you need to drop the rex part. Basically define it like EXTRACT-my_field = (?<error>\%.*-\b([0-4])\-.*?):\s

[MasterOogway]

Ok....I got it figured out. The Transorms and Props.conf files need to be on the same server. In this case it is the Stand-along Search server. Now I need to figure out a way to match any of the REGEX 'rex "(?\%.-\b([0-4])-.?):\s" ' to the list of ~900 events. The EXTACT-my_field = did nothing for me that I can see.

[MasterOogway]

Yes...you interpreted correctly. I am just trying to get over the hurdle on my first attempt at Lookup tables. Once I do I should be good.
My issue now is getting the search to recognize Lookup data. I have added "filename=syslog_alerter.csv" to the Search heads transforms.conf & added the "EXTRACT-cisco_event = \%.-\b([0-4])-.?):\s" to the Indexing servers Props.conf. A search doesn't complete and I get this: "Error in 'lookup' command: Could not find all of the specified lookup fields in the lookup table."

Are the correct files put on the correct index/search heads?
Thanks! ftk

[ftk]

I updated the answer, have a look.

[MasterOogway]

I understand the logic if you are expect <> "true", but how would I deal with a need to have the search look at all 900+ events and if no matches occur then email a group. Basically a "false" statement. The kicker is each search needs to check for both, true or false, and act on whichever it finds. ex. If event search matches one of the 900+, email group A OR If event search doesn't match one of the 900+, email group B.

[MasterOogway]

Thanks FTK.
I believe this is the exact track to take. On this same vein of thinking I would need to expand the column by one to look like this:

message, device, expected
SYS.-2-MOD_RESET, corp-core-rtf1, true
C4K_IOSMODPORTMAN-4-POWERSUPPLYGOOD, tsh-trhdist, true
SECURITY.-1-PORTSHUTDOWN, , true
DUAL.*-5-NBRCHANGE, 10.22., true
SYS-4-CHUNKSIBLINGSEXCEED, , true

How would you go about adding the 2nd column to make the qualifying statement true?

sourcetype=syslog_info | lookup my_lookup.csv my_field AS message AS device OUTPUT expected | where expected <> "true"

[ftk]

sorry i mean extract at search time, not index time.

[ftk]

I think you are on to something with using a lookup. To make that easier I recommend extracting the values you want to compare against as a field at index time, this will make comparisons a lot easier and you don't have to waste cycles on rex every time you do a search. You can use the regex you're using in rex to set up a field extraction, and then compare against your lookup table. I am editing my answer to go into a bit more detail on how I would set this up.

[MasterOogway]

...contintued...

This might be best suited to use a "lookup" table to compare against instead of a subsearch. If I find any -[0-4]-
then look it up against the table and perform the required act if found...if not found, email the default group.

Does this make more sense or simply cloud it up. What do you think about the Lookup table?

[MasterOogway]

The logic behind this is complex and I only gave a single sample of how a search "might" be completed.
Let me see if I can explain in more detail without confusing the matter.
I need to match any of 900+ events, many look like this:

SYS.-2-MOD_RESET
C4K_IOSMODPORTMAN-4-POWERSUPPLYGOOD
SECURITY.-1-PORTSHUTDOWN
SYS-3-PORT_DOT3_ALIGNMENT_ERROR
.
.
etc.
and if it doesn't match from that list of 900+ then an email is sent to a default group. What all the
events have in common is -[0-4]-. So my intent was to search for -[0-4]- and then look for all the potential combinations from the 900+.