trace a transaction through different fields

[jeremiahc4]

I've been poking around at this for a bit now to no avail. I'm sure it's something super simple and I'm just missing it.

I've got logs where one service has a trace ID that gets stored as the correlation ID in the next service. This can nest several layers deep as well. i.e.

Svc1 calls Svc2 with a trace ID

Svc2 stores svc1's trace ID as its correlation ID, creates it's own trace ID and calls Svc3

Svc3 stores svc2's trace ID as its correlation ID, creates it's own trace ID and calls SvcN

...

SvcN stores svc3's trace ID as its correlation ID, does some work and begins returning back up the stack

I need to be able to correlate these all together into one transaction from svc1-svcn. I have both the TraceID & CorrelateID extracted from every message. I'm able to do a simple transaction TraceID to bring together all messages that each svc generates, but I can't seem to correlate them between the two fields to create the full nested transaction.

[jeremiahc4]

Circling back just in case anyone else comes across this same requirement. What I wound up using was makemv to create a multivalue field, filter out some fluff using mvfilter, then do a transaction on that field.

index=test | eval parchild=parentId." ".childId | makemv parchild delim=" " | eval parchild=mvfilter(parchild!="NA" AND parchild!="null")| transaction parchild

I'm posting this several months after, so I can't recall if someone helped me to this or if I stumbled upon it myself (even a blind squirrel finds a nut sometimes). If you were the one that helped me find it, then say so in a comment and I'll award points.

Kristian, what your answer above misses is where the value of the second variable becomes the value of the first and the second gets a new value (which subsequently becomes the value of the first again).

[kristian_kolb]

It'd probably be easier if you pass the traceID along with the same name. Or extract it as such.
Possibly also generate new ID's for each level of nesting, like so;

Pseudo log events from these services:

svc1: call=svc2 SessionID=123 blah=hepp
svc2: call=svc3 SessionID=123 TransactionID=456 hepp=hupp
svc3: call=svc4 SessionID=123 TransactionID=456 RequestID=789 bleh=doh

Then you can make transactions on each and every layer. In the example above, I believe that you can skip the SessionID for the last entry, so that the transaction as a whole can be tied together with the SessionID between 1 and 2, and with the TransactionID between 2 and 3, but in 2, you need to have both SessionID and TransactionID to be able to connect 1 and 3 to the same transaction.

---

UPDATE:

I think that you could just extend that a little bit;

index=test | transaction traceid correlationid | ...

From the docs on transaction:

Splunk does not necessarily interpret the transaction defined by multiple fields as a conjunction (field1 AND field2 AND field3) or a disjunction (field1 OR field2 OR field3) of those fields. If there is a transitive relationship between the fields in the fields list, the transaction command will use it. For example, if you searched for

... | transaction host cookie

you might see the following events grouped into a transaction:

event=1 host=a
event=2 host=a cookie=b
event=3 cookie=b

/K

[jeremiahc4]

So based on what I show above, I am doing the following to try and follow a transaction;

index=test | transaction TraceID | search 0x123456

This gets them the first and second message, but not the third. (I use transaction as there are sometimes multiple lines with the same TraceID as it can call multiple async services)

[jeremiahc4]

Here's a couple example lines;
[code]
PERF,2013-04-03 11:29:52.640,external,0x123456,NA,service1,MyAPP,jimmy,NA,336,NA,NA
INFO,2013-04-03 11:29:53.189,internal,789012,0x123456,service2,TheirApp,jimmy,NA,174,NA,NA
INFO,2013-04-03 11:29:52.892,internal,345678,789012,service3,TheirApp,jimmy,NA,163,NA,NA
[/code]

[jeremiahc4]

I'm not sure I'm following your response. I should have included some example log messages I suppose as well as a key to the fields.
Here's my extract (I already know there's a better way for this);
EXTRACT-MYextract = (?i)^(?P[^,]\w+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+)

char limit, will post example logs in sep post