Splunk Search

How to create a search to find the largest contributor to increasing values?

las
Contributor

I have a log with statistics from mq containing some key values (time, MQmanager, Queuename) and some variables (number of messages added, number of messages removed and largest queue size).

If I do a search over time

sourcetype=MQResetStat (mqmanager=MQP) queue=* | dedup _raw | timechart sum(deqcount) 

I see a line where over time, more and more messages are pushed thru the system.

How do I create a search that tells me which queues are responsible for the increased number of messages?

I could of course just add by queue to the above search, but either the rise is hidden in "other" or there are to many queues to make the graf readable.

Ideally I would want just the 10 queues with the largest difference between the start and end period of the search, but still graphed over time with the sum of deqcount as the variable.

I could also do a subsearch, where I find the 10 queues, and then do the original search on top of that, but that might not be feasable due to time limit on subsearches.

I have tried looking for something similar, but apparently is not able to frame my search, so anything useable surfaces.

Any help is much appreciated.

kind regards

1 Solution

sundareshr
Legend

@somesoni2 had a great solution to a similar question last week here https://answers.splunk.com/answers/402954/how-to-use-top-in-timechart.html#answer-402996. Using that as the basis, I believe this should work in your case.

sourcetype=MQResetStat (mqmanager=MQP) queue=* | bin span=15m _time as time | stats sum(deqcount) as deqcount by time queue | eventstats first(deqcount) as start last(deqcount) as end by queue | eval total = end-start | sort - total | streamstats current=f window=1 max(total) as prevval by queue | eval toplist=case(isnull(prevval ),1,prevval =total,0,1=1,1) |accum toplist  | where toplist<10 | table time queue deqcount | xyseries time queue deqcount 

View solution in original post

sundareshr
Legend

@somesoni2 had a great solution to a similar question last week here https://answers.splunk.com/answers/402954/how-to-use-top-in-timechart.html#answer-402996. Using that as the basis, I believe this should work in your case.

sourcetype=MQResetStat (mqmanager=MQP) queue=* | bin span=15m _time as time | stats sum(deqcount) as deqcount by time queue | eventstats first(deqcount) as start last(deqcount) as end by queue | eval total = end-start | sort - total | streamstats current=f window=1 max(total) as prevval by queue | eval toplist=case(isnull(prevval ),1,prevval =total,0,1=1,1) |accum toplist  | where toplist<10 | table time queue deqcount | xyseries time queue deqcount 
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...