want to compare fields on different event and display only not matching fields

[mahson1]

Hi Team,

I have two events
1)
2017-05-18 14:24:58,798 [http-abcd] INFO Backend.Pure.gen.id - 108; Return 200 ids of type EID32.PROD for request aba6794f . Providing took 1019 ms

2017-05-09 11:54:10,651 [http-abcd] INFO Backend.Pure.gen.id - 110; Confirming ids for request aba6794f was successful

I want to match the request ID - aba6794f from both the events and wants to display only "IDs" and "Return 200" for which 2nd event has not occurred?

Any help?

[DalJeanis · Accepted solution]

Probably something like this.

your search that returns this type of event
| rex "(?<return>Return \d+)"
| rex "request\s+(?<request>\w+)" 
| rex "Providing took\s+(?<duration>\d+)"
| stats count as recCount, values(return) as return, values(duration) as duration by request

Add a | rex "(?<thing> pattern for thing)" and a values(thing) as thing for each additional thing you want to extract. (Not sure what you meant by IDs.)

If you want only unmatched, then you can add at the end...

| where recCount<2

...or...

| where isnull(thing) 

... when thing is something that would have only been on the record you expect to be missing.

[mahson1]

this is the query I am using and in that when I am matching the fields = splunk does not match with the list..

any solution over there?

host=abcd "Providing took" | rex field=_raw "request\s(?[^.\s]+)" | dedup ReqP | stats by ReqP
| appendcols [search host=abcd "IdGenEndpoint - 110; Confirming ids" | rex field=_raw "request\s(?[^\s+w\s]+)"
| dedup ReqC ]
| eval ReqPP=tostring(ReqP) | eval ReqCC=tostring(ReqC) | eval ReqCC1=rtrim(ReqCC, ".") | dedup ReqCC1
| eval Status = if(match(ReqPP,ReqCC1 ), "MATCH", "NO MATCH")
| table ReqPP, ReqCC1, Status, _time

need something like vlookup

sample ReqPP - 03af9a57-7820-4ff8-b78d-370cdffdbafd

[DalJeanis]

1) I've never yet seen a good use case for appendcols that something else wouldn't be better for. Appendcols just slams its return values onto the end of the other, one by one, without necessarily lining up the results based on the data. Avoid avoid avoid.

Where possible, just select all the relevant records, process them through logic that works individually for each type of record (leaving nulls where a field is not available or a value is not applicable, and then use stats to roll them together with list or values as appropriate.

When testing, keep lots of extra values around and test line by line until you know it's working, then you can then strip out the unneeded variables (if they have no long-term logical purpose), or just put a table command into the logic that doesn't pass them on after they have served their purpose.

Try this -

host=abcd ("Providing took" OR "IdGenEndpoint - 110; Confirming ids" )
| rex field=_raw "request\s*?(?<reqID>[^\.\s]+)" 
| rex field=_raw "Providing took\s*?(?<duration>\d+)" 
| eval reqtype=if(isnull(duration),"110","108")
| stats max(_time) as maxtime, min(_time) as mintime, list(_time) as alltimes, count as reccount, dc(reqtype) as reqsfound, values(reqtype) as reqtype, values(duration) as duration by reqID
| eval _time = maxtime
| eval Status = if(reqsfound>1,"MATCH","NOMATCH")

2) When posting code, please mark it as code, so that the stuff in angle brackets - extracted field names, etc - won't be eaten by the interface.

3) What the rex code [^.\s]+ reads as is "any character that is not any character and is not a space character, and as many additional characters that match what I just said." If you mean not a space character and not a period, then you need to escape the period. (The period which matches any character does not match certain special characters and markers like word breaks and, under certain directives circumstances, line breaks and the beginning or end of the string, so the combination is technically valid.)

4) What the rex code [^\s+w\s]+ reads is, "any character that is not a space character, a plus, a word character or a (redundant) space character, and as many additional characters that match what I just said."

You can go over to regex101.com to test various regular expressions and see if they pull what you think they should. Certain changes will have to be made to the regex that works over there, when you bring it over to splunk, so that it will work in the place it arrives in splunk.
- in a rex command, put the entire regular expression into double quotes, and therefore any double quotes in the regular expression will need to be escaped.
- in a map command, where there is a rex or regex as part of the search= string that will be executed, you will need to escape them once (so they are the way they would look if executed directly in a splunk command, and then escape them again. If I recall correctly, a single double-quote in a regular expression ends up with 3 slashes in front of it. The first pass turns \\ into \ and \" into ", then the second turns the surviving \" into ".

[DalJeanis]

Given the timestamps and wording, the Return event is the 2nd event in sequence, which makes your question confusing to me. Could you give one more example input that is unmatched, and what you want the output to look like?