How to create a search with conditional field extraction logic? If sourcetype=x, use rex extraction 1, but if source=y, use rex extraction 2

[dbcase]

Hi,

I have the following search:

host="*beta*" index=wls OR index=main sourcetype=wls_managedserver OR source="/etc/httpd/logs/portal-access_log*" |rex field=message_text "UCE-(?< UCE_Code >[^\s\:;]+)"|rex "UCE-(?< UCE_Code1 >[^\"]+)"|table UCE_Code UCE_Code1

The search works if I break it apart, meaning using:

Index=wls and sourcetype=wls_managedserver with rex UCE-(?< UCE_Code >[^\s\:;]+)" as query1

index=main source="/etc/httpd/logs/portal-access_log*" with rex "UCE-(?< UCE_Code1 >[^\"]+)" as query2
but of course I'd like to combine the two searches. How can I have the rex definitions associate to a sourcetype/index/source?

[sundareshr · Accepted solution]

Try this run-anywhere sample. (use everything after the | table x segment)

| gentimes start=-1 | eval x="< Warning > < ucontrol > < betamax-cpe2 > < managedServer2 > < pool-5-thread-4 > << anonymous >> < > < > < 1471557920294 > < BEA-000000 > < fn.xmpp.v2.IQSMAPHandler - UCE-24100 - Server exception while processing SMAP message '/event/lighting' from '1000275@xmpp/9c972687947f'|

 - - [18/Aug/2016:17:07:22 -0500] \"GET /rest/icontrol/sites/1001226/network/instances/1004a1514a4f98.0/functions/getUserCodes HTTP/1.1\" 404 43 0 10418 \"-\" \"Mozilla/5.0 (Linux; Android 6.0.1; SAMSUNG-SM-G930A Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/52.0.2743.98 Mobile Safari/537.36\" \"UCE-16001\"|

 - - [18/Aug/2016:17:13:28 -0500] \"GET /rest/icontrol/users/1000251/preferences/scenes HTTP/1.1\" 404 46 0 22809 \"-\" \"Apache-HttpClient/4.3.6 (java 1.5)\" \"UCE-16000\""| makemv x delim="|" | mvexpand x | table x | rex field=x "UCE-(?<code>(\d+\")|(\d+\s.*))" | table x code | rex mode=sed field=code "s/(\d+)\"/\1/g"

[dbcase]

Hi Sundareshr,

I think you are very close. The table does have the UCE code (numeric digits) but it also has the error description which is pretty technical and won't be understood by the users. I've made a lookup table that replaces the description with a more user friendly one. How can the rex be modified so the UCE code is only the digits?

[dbcase]

Thanks (again) Sundareshr!!

[dbcase]

Think I figured it out UCE-(?< code >(\d+))

[sundareshr]

If all you need is the numbers, you just need this. You don't need the sed

... | rex field=x "UCE-(?<code>\d+) | ...

[sundareshr]

How about something like this.

host="*beta*" index=wls OR index=main sourcetype=wls_managedserver OR source="/etc/httpd/logs/portal-access_log*" | rex field=message_text "UCE-(?< UCE_Code >[^\s\:;]+)"| rex "UCE-(?< UCE_Code1 >[^\"]+)" | eval UCE_Code=if(source="/etc/httpd/logs/portal-access_log*", UCE_CODE1, UCE_CODE) | table UCE_Code

[dbcase]

Hi sundareshr, thats close! One problem though, the rex extraction for UCE_code1 is pulling the wrong data because it is looking at a different log than intended.

[dbcase]

a bit more detail.....

rex field=message_text "UCE-(?< UCE_Code >[^\s:;]+)" gets the right UCE Code if it is looking at index=wls sourcetype=wls_managedserver events

similarly rex "UCE-(?< UCE_Code1 >[^\"]+)" gets the right UCE Code if it is looking at index=main source="/etc/httpd/logs/portal-access*log*" events

but when the sourcetype/index/sources are cobbled together the rex gets confused because the formats don't match if you switch them.

[sundareshr]

I understand. I am wondering if there is a different rex that could be applied to make this work without two searches. Can you share couple of samples, you can obfuscate any sensitive info.

[dbcase]

rex field=message_text "UCE-(?< UCE_Code >[^\s:;]+)" index=wls sourcetype=wls_managedserver events

< Aug 18, 2016 5:05:20 PM CDT > < Warning > < ucontrol > < betamax-cpe2 > < managedServer2 > < pool-5-thread-4 > << anonymous >> < > < > < 1471557920294 > < BEA-000000 > < fn.xmpp.v2.IQSMAPHandler - UCE-24100 - Server exception while processing SMAP message '/event/lighting' from '1000275@xmpp/9c972687947f'

rex "UCE-(?< UCE_Code1 >[^\"]+)" index=main source="/etc/httpd/logs/portal-access*log*" events

"beta.icontrol.com" 99.98.192.121 "99.98.192.121" - - [18/Aug/2016:17:07:22 -0500] "GET /rest/icontrol/sites/1001226/network/instances/1004a1514a4f98.0/functions/getUserCodes HTTP/1.1" 404 43 0 10418 "-" "Mozilla/5.0 (Linux; Android 6.0.1; SAMSUNG-SM-G930A Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/52.0.2743.98 Mobile Safari/537.36" "UCE-16001"

"beta.icontrol.com" 54.174.106.18 "54.174.106.18" - - [18/Aug/2016:17:13:28 -0500] "GET /rest/icontrol/users/1000251/preferences/scenes HTTP/1.1" 404 46 0 22809 "-" "Apache-HttpClient/4.3.6 (java 1.5)" "UCE-16000"

[sundareshr]

Try this regex `"UCE-(?(\d+\")|(\d+\s.*))"


[sundareshr]

can you share a sample for each type (source="/etc/httpd/logs/portal-access_log*" AND the other)