As part of our index, we log events for every request we make to our downstream systems. Each system which receives a request appends a TraceContext (GUID) to the incoming TraceContext. Idea is to have a way to get the chain of events.
Here are sample set of event messages:
SourceName=QueryAPI
EventCode=11
Payload={
"TraceContext":"91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d",
"EventName":"Start"
}
SourceName=QueryAPI
EventCode=10
Payload={
"TraceContext":"91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d|5a595ffe-9a5d-4abd-93fb-d57c3f427af0",
"EventName":"Receive"
}
SourceName=QueryAPI
EventCode=9
Payload={
"TraceContext":"91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d|5a595ffe-9a5d-4abd-93fb-d57c3f427af0",
"EventName":"Send"
}
SourceName=QueryAPI
EventCode=12
Payload={
"TraceContext":"91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d",
"EventName":"Stop"
}
I am running the search below to identify the starting event TraceContext and using that in subsearch, hoping to see all the matching events, but my search does not produce any events:
index=IndexName SourceName=SomeName [search index=IndexName SourceName=SomeName EventCode=11 | table TraceContext]
If I just run the subsearch search separately and use the TraceContext string and use that for a new search, I can find all the matching events.
index=IndexName SourceName=SomeName EventCode=11 | table TraceContext
-> returns "91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d"
and running
index=IndexName SourceName=SomeName "91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d"
returns all records. I am not sure what am I doing wrong.
Try this:
index=IndexName SourceName=SomeName [search index=IndexName SourceName=SomeName EventCode=11 | return $TraceContext]
I have had the same thing happen to me in the past. I opened a ticket with Splunk and they didn't know what to make of it either. The funny thing is, it worked fine on one Search Head and did not work on another.
Try this:
index=IndexName SourceName=SomeName [search index=IndexName SourceName=SomeName EventCode=11 | return $TraceContext]
TraceContext is not an indexed field. It is part of a JSON string, which is indexed as Payload field.
by default return statement only returns the first matching event. However we can specify the count also. I modified the query as below and I am getting required results now:
index=IndexName SourceName=SomeName [search index=IndexName SourceName=SomeName EventCode=11 | return 100 $TraceContext]
This works fine but it only returns events for one TraceContext. I would like to to get all events matching all traceContext available in the index.
Do all events have a field named TraceContext
? If so, then you can do this:
index=IndexName SourceName=SomeName [search index=IndexName SourceName=SomeName EventCode=11 | fields TraceContext]