Can't use rex extracted value in new search because value isn't an extracted field

[Whistler]

Hi all,

I've written the following query:

sourcetype=mysourcetype DA-bericht [search sourcetype=mysourcetype "Bericht versturen van 'XXX' naar 'YYY'" | fields + KetenId] | rex field=_raw "(?m).*aanvraagvolgnr/versie:(?<volgnrversie>.*?),.*" | fields + volgnrversie | table volgnrversie

What I want to do is grab all messages which contain the tekst "Bericht versturen van 'XXX' naar 'YYY'" and then select the KetenId field (this is a correlation ID) to use in a second search. That second search should result in all messages which contain the text "DA bericht" and the KetenId. From those messages, I want to extract another kind of ID into the variable 'volgnrversie'. This all seems to work fine and if I redirect the output to a table I see the results I expected. For example:

• 700361886/1
  
  • FIGLO-EA1C4235-9EEA-FFA9-3BED-22DE4373BD6C-2DBF17BFB641426A9908A2FF579C231A-201609091353/1
  • ISW-8A2F62B1-7C26-4E37-858C-09BC2DC35577/1

The problem arises when I want to use the above results in yet another search:

sourcetype=mysourcetype [search sourcetype=mysourcetype DA-bericht [search sourcetype=mysourcetype "Bericht versturen van 'XXX' naar 'YYY'" | fields + KetenId] | rex field=_raw "(?m).*aanvraagvolgnr/versie:(?<volgnrversie>.*?),.*" | fields + volgnrversie]

This doesn't result in anything. If I change the query and hardcode on of the above results into it, then it does provide many results.

sourcetype=mysourcetype 700361886/1

Can anyone explain what I'm doing wrong? Thanks!

The jobs section shows me the inner searches do provide events:

9/15/16 1:21:49 PM whistler search 5.20MB 0 00:00:44 Sep 15, 2016 1:32:35 PM Done Inspect | Save | Delete

"mysearch" [earliest time=9/8/16 1:00:00 PM, latest time=9/15/16 1:21:49 PM]

9/15/16 1:21:49 PM whistler search 1.64MB 3,554 00:00:07 Sep 15, 2016 1:26:57 PM Done Inspect | Save | Delete

| subsearch [earliest time=9/8/16 1:00:00 PM, latest time=9/15/16 1:21:49 PM]

9/15/16 1:21:49 PM whistler search 0.43MB 3,555 00:00:01 Sep 15, 2016 1:26:51 PM Done Inspect | Save | Delete

| subsearch [earliest time=9/8/16 1:00:00 PM, latest time=9/15/16 1:21:49 PM]

When inspecting the last job (the main search) then I do see the following at 'normalizedSearch': litsearch sourcetype=mysourcetype ( ( volgnrversie="FNDSK-EGS4OUITDT5UPX63GOEQ-1-AX-AA/1" ) OR ( volgnrversie="B1-00001.7011614#07072016191529/1" ) OR ( volgnrversie="FNDSK-E2N6YALZ7AOTYCKWL52A-5-AX-AG/1" )

Could that be the problem? The volgnrversie isn't an extracted field, but something I grabbed from the search results?
Hm, appears so. When I change the query to the following I do get results:

sourcetype=mysourcetype [search sourcetype=mysourcetype DA-bericht [search sourcetype=mysourcetype "Bericht versturen van 'XXX' naar 'YYY'" | fields + KetenId] | rex field=_raw "(?m).*aanvraagvolgnr/versie:(?<volgnrversie>.*?),.*" | fields + KetenId]

However, I can't use the KetenId. I need the volgnrversie variable to continue my search. How do I get this to work?

With regards,

Reinier

[sundareshr]

Try this

 sourcetype=mysourcetype [search sourcetype=mysourcetype DA-bericht [search sourcetype=mysourcetype "Bericht versturen van 'XXX' naar 'YYY'" | fields + KetenId] | rex field=_raw "(?m).*aanvraagvolgnr/versie:(?<search>.*?),.*" | fields search ]

Although you may be able to get the same results without using sub-searches.

[Whistler]

Hi, thanks for the response. Unfortunately that also doesn't yield any results. As far as I can see you removed the plus sign from the query. By default splunk will use the + if nothing is specified:

Keeps (+) or removes (-) fields from
search results based on the field list
criteria. If + is specified, only the
fields that match one of the fields in
the list are kept. If - is specified,
only the fields that match one of the
fields in the list are removed. If
neither is specified, defaults to +.

[sundareshr]

From the Job Inspector, you mentioned, the base search does not have the field volgnrversie extracted, so a query volgnrversie =xyz will return nothing. I rename the field to search in the last sub-search, so the base search will see ( ( "FNDSK-EGS4OUITDT5UPX63GOEQ-1-AX-AA/1" ) OR ( "B1-00001.7011614#07072016191529/1" ) ),

Can you confirm you see this in the Job Inspector.

[Whistler]

Edit: The solution, renaming the variable to search, doesn't seem to be working as well as I thought. It appears that the search is based only 1 of the results from the sub-search.

At first I saw results and thought the new search was working, but when analysing the results I noticed there were too few. Then I did what you asked (sorry) and looked at the Job inspector. It show something like "litsearch sourcetype=mysourcetype FNDSK-EGS4OUITDT5UPX63GOEQ-1-AX-AA/1". Not the many OR statements it was using earlier.

[sundareshr]

If all the sourcetypes are the same, try this (without sub-searches)

sourcetype=mysourcetype | eval match=if(searchmatch("Bericht versturen van 'XXX' naar 'YYY'"), "yes", null()) | rex field=_raw "(?m).*aanvraagvolgnr/versie:(?<volgnrversie>.*?),.*" | where isnotnull(volgnrversie) AND match="yes"

***If, they are not, try this

sourcetype=mysourcetype1 OR sourcetype=mysourcetype2 OR sourcetype=mysourcetype3 | eval match=if(sourcetype=mysourcetype2 AND searchmatch("Bericht versturen van 'XXX' naar 'YYY'"), "yes", null()) | rex field=_raw "(?m).*aanvraagvolgnr/versie:(?.*?),.*" | where isnotnull(volgnrversie) AND sourcetype=mysourcetype3

[Whistler]

I didn't get that to work. I gave up and finally just added the aanvraagvolgnr/versie as a data extraction. My first query gave the correct results after that. There is a Splunk training at my company soon. I'll try to rewrite the query at that session. If that results in anything useful, I'll post it here.