Concurrency Command Appears not to work as expected.

[bischofk]

Here is my query:

index=dotcom source=*systemout.log eventtype=performance *StoreInventoryTransport |
transaction thread_id host startswith="BEGIN StoreInventory SERVICE CALL" endswith="END StoreInventory SERVICE CALL" |
concurrency duration=duration |
timechart avg(duration) max(concurrency) by host

What this query is meant to do is capture transactions of some outbound service calls our application server is making. Example transaction results look like so:

» 11/16/12
9:06:00.401 AM

[11/16/12 9:06:00:401 EST] 0000006c IG E com.nad.integration.transport.sbd.StoreInventoryTransport sendSynchronous() Store ID : 10001 : BEGIN StoreInventory SERVICE CALL [PERFORMANCE]
[11/16/12 9:06:01:061 EST] 0000006c IG E com.nad.integration.transport.sbd.StoreInventoryTransport sendSynchronous() Store ID : 10001 : END StoreInventory SERVICE CALL [PERFORMANCE]

host=sdmpras02   Options|  
sourcetype=dotcom_system_out   Options|  
source=/usr/WebSphere/AppServer764/profiles/WCSDM64/logs/SVR02-P/SystemOut.log   Options

2 » 11/16/12
9:06:00.658 AM

[11/16/12 9:06:00:658 EST] 0000004a IG E com.nad.integration.transport.sbd.StoreInventoryTransport sendSynchronous() Store ID : 10001 : BEGIN StoreInventory SERVICE CALL [PERFORMANCE]
[11/16/12 9:06:02:099 EST] 0000004a IG E com.nad.integration.transport.sbd.StoreInventoryTransport sendSynchronous() Store ID : 10001 : END StoreInventory SERVICE CALL [PERFORMANCE]

host=sdmpras09   Options|  
sourcetype=dotcom_system_out   Options|  
source=/usr/WebSphere/AppServer764/profiles/WCSDM64/logs/SVR09-P/SystemOut.log   Options

The transaction results are perfectly fine. However, the problem is that the concurrency command appears to be crossing the boundary of the host. I have roughly 20 hosts in the cluster, and I want to know the concurrency of these transactions WITHIN the host, not across the hosts. When I chart the max(concurrency) like shown in my query, I get the concurrency of that Host's transactions considering the transactions on the other hosts as well, essentially givng me a concurrency per host that is 20x higher than I want (essentially total concurrency).

Please help 🙂

[bischofk]

Yeah, adding a concurrency by host was my first try :). Perhaps they should add that!

Otherwise, I am thinking I would need to calculate the concurrency by dividing the number of transactions by the time span in Timechart. I am now having trouble attempting that as well, because I want to remove fields that are used to do the math ( | fields - transactionCount ), but that doesnt appear to work with timechart...ARGH!!!

[sdaniels]

This approach might be able to give you the concurrency count you are looking for by host:

http://splunk-base.splunk.com/answers/7269/how-to-calculate-concurrent-transactions-grouped-with-a-p...

You won't get a list of the transactions by host however since concurrency does not have a 'by' clause.