Splunk Search

Calculate time difference in two different logs

cj039165
New Member

Hello -

Stumped on this. I have two different log files. One logs the time (and data) in transactions sent, the other has the time (and data) received. I would like to calculate the 'response' time. From there we could could alert if it goes above a set period of time. Each transactions is link between the two logs via a thread id. I've very new to Splunk, any help would be great.

Log example Send:
10:31:55,462 DEBUG [Thread-216189]

Log example Receive:
10:32:03,067 DEBUG [Thread-216189]

0 Karma

cj039165
New Member

Noticed a problem with the search results. It turns out it's possible to have the same thread-id in on multiple servers. For example [Thread-216189] can be in UNITED_Send.log / UNITED_Receive.log on HOST WASAPP07 or WASAPP08 or WASAPP09. How do I get it to not calculate across the hosts?

Search String:

index=hdx_payer source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/UNITED_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/UNITED_Send.log" | eval Time=_time | stats values(Time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=tonumber(mvindex(Time,1))-tonumber(mvindex(Time,0)) | convert ctime(Time)
0 Karma

apilger_splunk
Splunk Employee
Splunk Employee

Try this:

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | stats min(_time) AS my_start,max(_time) AS my_end by  thread_id| eval reponse_time=my_end-my_start
0 Karma

sundareshr
Legend

Try transaction

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | transaction thread_id | rename duration AS response_time | table thread_id response_time 
0 Karma

cj039165
New Member

Ran the search. I don't get an error, however I don't see any search results. Do I need to extract [thread-345432] into a extracted field?

Thanks

0 Karma

sundareshr
Legend

You will have to extract the thread_id field. Try this (you may need to correct sent & received as well)

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | rex "Thread-(?<thread_id>\d+)" | transaction thread_id startswith="sent" endswith="received" | rename duration AS response_time | table thread_id response_time 
0 Karma

somesoni2
SplunkTrust
SplunkTrust

Just include the host field in the transaction command

0 Karma

cj039165
New Member

Tried it yesterday. For some reason that did not work. It appears adding host=* shows the right data. Will review with the group looking for the info. Thanks.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try like this (check the field name)
Updated

(base search for log 1 - transaction sent ) OR (base search for log 2 - transaction received) | stats values(_time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=tonumber(mvindex(Time,1))-tonumber(mvindex(Time,0)) | convert ctime(Time)

If above still doesn't work, try this

(base search for log 1 - transaction sent ) OR (base search for log 2 - transaction received) | eval Time=_time | stats values(Time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=tonumber(mvindex(Time,1))-tonumber(mvindex(Time,0)) | convert ctime(Time)
0 Karma

cj039165
New Member

Hello. Thanks for the response. I ran the following search. I'm getting the following error: Error in 'eval' command: Typechecking failed. '-' only takes numbers.

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | stats values(_time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=mvindex(Time,1)-mvindex(Time,0) | convert ctime(Time)
0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try the updated answer.

0 Karma

cj039165
New Member

I did your second search after extracting the thread-id. I'm see what I needed. Thanks very much for the help!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...