I an new to Splunk and amdeveloping it to provide monitoring of message flow across our landscape and would like to know how i can set a threshold, apply it to a field and then alert out on a dashboard if that threshold is breached.
Use case is a a message is flowing across between applications, if a transient step in between has a build up of messages in it that breach a pre defined threshold then display an alert on a dashboard which is monitored by our operatiosn group.
Thanks in advance
Nick
There's two things potentially mixed up here: Displaying a breach of threshold on a dashboard, and triggering an alert based on a breach of threshold. From a search language point of view both are basically the same thing, but where you use them within Splunk isn't.
For the dashboard way, you could have a search like this:
search for your events with queue sizes | stats max(queue_size) as max_queue_size | rangemap field=max_queue_size low=0-9 elevated=10-19 default=severe
Use that search in a single value panel like this:
<single>
<search>
<query>see above</query>
</search>
<option name="classField">range</option>
</single>
That'll display the maximum queue size in a colour determined by the rangemap - there are classes pre-defined in CSS to magically do this.
The alert is a bit different, you save a search like this as an alert:
search for your events with queue sizes | stats max(queue_size) as max_queue_size
Then set it to execute on some schedule with some matching time range, for example to run every hour with -61m@m
to -m@m
as time range to allow for up to one minute of delay, and define the alert's trigger condition as where max_queue_size > 19
if it should alert above 19.