Splunk Search

Transaction very slow

iberecamara
Engager

I have to calculate the response time from an application that depends on the response of another application. For that, I need to get the response times from both and calculate. The response time from Application1 and Application2 are in different places, but they share a couple of unique fields. I'm using a transaction for this right now, but its being very very slow, can someone help me refactor this? The search string is below, thanks!

index=myindex source=mysource sourcetype=mysourcetype host=myhost CLASS=PERFORMANCE_LOG (PERF_TYPE=App1PerformanceMetrics OR PERF_TYPE=App2PerformanceMetrics)

| transaction TXN_ID COR_ID connected=true
| search eventcount=2 status=SUCCESS
| eval responseTime = app1ResponseTime - app2ResponseTime

| timechart avg(responseTime)

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Is combination of TXN_ID COR_ID unique for all transactions or they may overlap (there multiple records with combination of TXN_ID COR_ID that may exist at different time)?

0 Karma

iberecamara
Engager

They are unique, or at least should be

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Give thisa try

index=myindex source=mysource sourcetype=mysourcetype host=myhost CLASS=PERFORMANCE_LOG (PERF_TYPE=App1PerformanceMetrics OR PERF_TYPE=App2PerformanceMetrics) 
| stats min(_time) as _time count as eventcount values(status) as status values(app1ResponseTime) as app1ResponseTime values(app2ResponseTime) as app2ResponseTime by TXN_ID COR_ID connected=true 
| search eventcount=2 status=SUCCESS 
| eval responseTime = app1ResponseTime - app2ResponseTime 
| timechart avg(responseTime)

iberecamara
Engager

I just had to remove the connected=true, but it seems that it works. I'm in the middle of a meeting right now, but I'll make some tests when I can, and if it doesn't break for different time ranges, I'll accept this as the answer, thank you very much!

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

@iberecamara,

Please try this and lets know if it gives same result and where's the difference.

index=myindex source=mysource sourcetype=mysourcetype host=myhost CLASS=PERFORMANCE_LOG (PERF_TYPE=App1PerformanceMetrics OR PERF_TYPE=App2PerformanceMetrics) 
|stats values(app1ResponseTime) as app1ResponseTime,dc(PERF_TYPE) no_of_perf_type,latest(status) as status by TXN_ID,COR_ID
|where no_of_perf_type>1 AND status="SUCCESS"
|eval app1ResponseTime=mvindex(app1ResponseTime,0), app2ResponseTime=mvindex(app2ResponseTime,1)|eval responseTime = app1ResponseTime - app2ResponseTime 
|timechart avg(responseTime)
Happy Splunking!
0 Karma

iberecamara
Engager

@renjith.nair
It errors trying to run the search.

Error in 'stats' command: The aggregation specifier 'dc(PERF_TYPE) no_of_perf_type' is invalid. The aggregation specifier must be in func_name format.

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

sorry, missed as between them . Replace it by dc(PERF_TYPE) as no_of_perf_type

Happy Splunking!
0 Karma

iberecamara
Engager

The results are different from mine. My search is returning a couple matches and generating a line chart as intended, but this search returns "No results found."

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

do you get some results for this?

 index=myindex source=mysource sourcetype=mysourcetype host=myhost CLASS=PERFORMANCE_LOG (PERF_TYPE=App1PerformanceMetrics OR PERF_TYPE=App2PerformanceMetrics) 
 |stats values(app1ResponseTime) as app1ResponseTime,dc(PERF_TYPE) no_of_perf_type,latest(status) as status by TXN_ID,COR_ID
Happy Splunking!
0 Karma

iberecamara
Engager

Yes, I do. It gives me the app1ResponseTime, no_of_perf_type and status for the matches in TXN_ID and COR_ID

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

sorry missed one field,
index=myindex source=mysource sourcetype=mysourcetype host=myhost CLASS=PERFORMANCE_LOG (PERF_TYPE=App1PerformanceMetrics OR PERF_TYPE=App2PerformanceMetrics)
|stats values(app1ResponseTime) as app1ResponseTime,values(app2ResponseTime) as app2ResponseTime,dc(PERF_TYPE) no_of_perf_type,latest(status) as status by TXN_ID,COR_ID
|where no_of_perf_type>1 AND status="SUCCESS"
|eval responseTime = app1ResponseTime - app2ResponseTime
|timechart avg(responseTime)

Happy Splunking!
0 Karma

auraria1
Path Finder

Can you move the search eventcount=2 status=SUCCESS into the main query or is that info dictated by the transaction?

That should help speed due to the fact you'll be transacting more logs, then performing a search on that.

Searching is a lot less resource intensive than transaction, so if you limit the amount of results prior it should help the transaction speed as well.

Does that make sense?

0 Karma

iberecamara
Engager

I know that I should narrow as much as possible before doing the transaction, but unfortunately no. The eventcount is a direct effect of the transaction, and the status=SUCCESS is only present in App1PerformanceMetrics.

0 Karma

auraria1
Path Finder

gotcha, is there anything else that you can filter on that you're specifically looking for in the main search?

Also you can remove the parens around the PERF_TYPE OR search. Don't think that'll make a difference though.

Are these going to a specific index you can filter on? Generally doing index and sourcetype/host is faster.

0 Karma

iberecamara
Engager

I'm looking for response times from App1PerformanceMetrics and App2PerformanceMetrics, so I can eval them later for the real response time I want (app1ResponseTime - app2ResponseTime). I know the parens don't help at all, I like to use them as a visual aid.
I'm already using an index, I updated the search here.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...