Splunk Search

How to use Transaction and the sum of the same name field in a rangemap and run stats?

slatta
Explorer

Trying to use the sum of "docCount" in a transaction and use that value for the range and then run statistics by the range. The transaction is working as expected and get the correct logs entries. If I just run the command without worrying about the sum of docCounts the count and stats work as expected, but only get a docCount max of 50. When adding the sum(docCount), the count sums, but lose the ability to run stats on the "duration" field from the log.

//Range counts properly, but no stats

index=ts* Component=TS ("StartingJob" OR "Starting module=Commit" OR "FinishedJob") | transaction TSJobId startswith="StartingJob" endswith="FinishedJob" keepevicted=false | stats sum(docCount) as doccounter by TSJobId|rangemap field=doccounter "0-5Docs"=0-5, "6-10Docs"=6-10, "11-20Docs"=11-20, "21-30Docs"=21-30, "31-40Docs"=31-40, "41-50Docs"=41-50, "51-60"=51-60, "60-80"=60-80, default=">80Docs" | stats count, avg(duration), p75(duration), p80(duration), p90(duration), p95(duration) by range

//Stats work, but missing the actual docCount

index=ts* Component=TS ("StartingJob" OR "Starting module=Commit" OR "FinishedJob") | transaction TSJobId startswith="StartingJob" endswith="FinishedJob" keepevicted=false | rangemap field=doccounter "0-5Docs"=0-5, "6-10Docs"=6-10, "11-20Docs"=11-20, "21-30Docs"=21-30, "31-40Docs"=31-40, "41-50Docs"=41-50, "51-60"=51-60, "60-80"=60-80, default=">80Docs" | stats count, avg(duration), p75(duration), p80(duration), p90(duration), p95(duration) by range

Result of Transaction command (Important Fields: TSJobId, docCount, duration):

[2015-08-07 14:31:06,220] DEBUG Component=TSThread="DefaultMessageListenerContainer-9" TSJobId=20150807_143106_5910 MessageId= REQ_ID=94700114 StartingJob tsJobType=ingest sync=false
[2015-08-07 14:31:07,614] DEBUG Component=TSThread="DefaultMessageListenerContainer-9" TSJobId=20150807_143106_5910 MessageId= REQ_ID=94700114 Starting module=Commit docCount=7
[2015-08-07 14:31:07,614] DEBUG Component=TSThread="pool-2-thread-27" TSJobId=20150807_143106_5910 MessageId= REQ_ID=94700114 Starting module=Commit docCount=50
[2015-08-07 14:31:07,614] DEBUG Component=TSThread="pool-3-thread-20" TSJobId=20150807_143106_5910 MessageId= REQ_ID=94700114 Starting module=Commit docCount=50
[2015-08-07 14:31:31,223] DEBUG Component=TSThread="DefaultMessageListenerContainer-9" TSJobId=20150807_143106_5910 MessageId= REQ_ID=94700114 FinishedJob tsJobType=ingest duration=25004

[2015-08-07 14:26:07,646] DEBUG Component=TSThread="DefaultMessageListenerContainer-7" TSJobId=20150807_142607_5888 MessageId= REQ_ID=94696492 StartingJob tsJobType=ingest sync=false
[2015-08-07 14:26:08,349] DEBUG Component=TSThread="DefaultMessageListenerContainer-7" TSJobId=20150807_142607_5888 MessageId= REQ_ID=94696492 Starting module=Commit docCount=23
[2015-08-07 14:26:28,867] DEBUG Component=TSThread="DefaultMessageListenerContainer-7" TSJobId=20150807_142607_5808 MessageId= REQ_ID=94696492 FinishedJob tsJobType=ingest duration=11221


[2015-08-07 14:25:03,846] DEBUG Component=TSThread="DefaultMessageListenerContainer-7" TSJobId=20150807_142503_5808 MessageId= REQ_ID=94696492 StartingJob tsJobType=ingest sync=false
[2015-08-07 14:25:05,343] DEBUG Component=TSThread="DefaultMessageListenerContainer-7" TSJobId=20150807_142503_5808 MessageId= REQ_ID=94696492 Starting module=Commit docCount=3
[2015-08-07 14:25:05,343] DEBUG Component=TSThread="pool-2-thread-26" TSJobId=20150807_142503_5808 MessageId= REQ_ID=94696492 Starting module=Commit docCount=50
[2015-08-07 14:25:05,343] DEBUG Component=TSThread="pool-3-thread-11" TSJobId=20150807_142503_5808 MessageId= REQ_ID=94696492 Starting module=Commit docCount=50
[2015-08-07 14:25:28,377] DEBUG Component=TSThread="DefaultMessageListenerContainer-7" TSJobId=20150807_142503_5808 MessageId= REQ_ID=94696492 FinishedJob tsJobType=ingest duration=24532

[2015-08-07 14:23:59,434] DEBUG Component=TSThread="DefaultMessageListenerContainer-6" TSJobId=20150807_142359_5814 MessageId= REQ_ID=94695906 StartingJob tsJobType=ingest sync=false
[2015-08-07 14:24:01,094] DEBUG Component=TSThread="DefaultMessageListenerContainer-6" TSJobId=20150807_142359_5814 MessageId= REQ_ID=94695906 Starting module=Commit docCount=1
[2015-08-07 14:24:01,095] DEBUG Component=TSThread="pool-2-thread-22" TSJobId=20150807_142359_5814 MessageId= REQ_ID=94695906 Starting module=Commit docCount=50
[2015-08-07 14:24:24,164] DEBUG Component=TSThread="DefaultMessageListenerContainer-6" TSJobId=20150807_142359_5814 MessageId= REQ_ID=94695906 FinishedJob tsJobType=ingest duration=24730
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Try something like this

index=ts* Component=TS ("StartingJob" OR "Starting module=Commit" OR "FinishedJob") | transaction TSJobId startswith="StartingJob" endswith="FinishedJob" keepevicted=false | eventstats sum(docCount) as doccounter by TSJobId| rangemap field=doccounter "0-5Docs"=0-5, "6-10Docs"=6-10, "11-20Docs"=11-20, "21-30Docs"=21-30, "31-40Docs"=31-40, "41-50Docs"=41-50, "51-60"=51-60, "60-80"=60-80, default=">80Docs" | stats count, avg(duration), p75(duration), p80(duration), p90(duration), p95(duration) by range

View solution in original post

lguinn2
Legend

In the second search, you use "doccounter" as the field for the rangemap, but the field does not exist. So the rangemap won't work in the second search. In the first search, you calculated "doccounter" before using it. As a debugging exercise: what do you get when you run -

index=ts* Component=TS ("StartingJob" OR "Starting module=Commit" OR "FinishedJob") 
| transaction TSJobId startswith="StartingJob" endswith="FinishedJob" keepevicted=false 
| stats sum(docCount) as doccounter by TSJobId
| sort -doccounter
0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try something like this

index=ts* Component=TS ("StartingJob" OR "Starting module=Commit" OR "FinishedJob") | transaction TSJobId startswith="StartingJob" endswith="FinishedJob" keepevicted=false | eventstats sum(docCount) as doccounter by TSJobId| rangemap field=doccounter "0-5Docs"=0-5, "6-10Docs"=6-10, "11-20Docs"=11-20, "21-30Docs"=21-30, "31-40Docs"=31-40, "41-50Docs"=41-50, "51-60"=51-60, "60-80"=60-80, default=">80Docs" | stats count, avg(duration), p75(duration), p80(duration), p90(duration), p95(duration) by range

slatta
Explorer

Changing from stats sum(docCount) to eventstats sum(docCount) seems to have made the difference. Thanks.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...