Splunk Search

How to alert on license pool violation and can i trigger a search upon the violation

a212830
Champion

Hi,

I'm referencing this post - https://answers.splunk.com/answers/321226/how-to-create-an-alert-to-notify-me-via-email-when.html

I want to alert on pool violations, and gives me the correct info, but I'm not sure where to put the threshold check. I've put the "where" clause in a number of places, and each time, it returns nothing.

Also, is there a way to trigger a search when this happens and then send an email from the results?

0 Karma

yoho
Contributor

@mmodestino_splunk, I've compared your search with the one in the DMC for the total license. This one seems to take into account you can have multiple license masters (splunk_server in the search). It also uses a join which is a bit more explicit than a "|search []" in my opinion. So I've rewritten partly your search and the result is below.

However, in both cases, you consider a license group can only contain 1 single stack (by renaming stack_ids to stack_id), is it really the case ?

Here's my search :

| rest splunk_server=local /services/licenser/pools 
| rename title AS pool 
| join type=outer splunk_server stack_id 
 [ rest splunk_server=local /services/licenser/groups 
 | eval stack_id=stack_ids
 | fields stack_id splunk_server is_active]
| search is_active=1
| eval quota=if(isnull(effective_quota),quota,effective_quota) 
| eval "% used"=round(used_bytes/quota*100,2) 
| fields pool "% used"
| where '% used' >= 0
0 Karma

mattymo
Splunk Employee
Splunk Employee

Hi a212830!

The where clause goes at the end, but you gotta be careful with fields with spaces...i used single quotes to get it to work.

| rest splunk_server=local /services/licenser/pools 
| rename title AS Pool 
| search 
    [ rest splunk_server=local /services/licenser/groups 
    | search is_active=1 
    | eval stack_id=stack_ids 
    | fields stack_id] 
| eval quota=if(isnull(effective_quota),quota,effective_quota) 
| eval "% used"=round(used_bytes/quota*100,2) 
| fields Pool "% used"
| where '% used' > .2

alt text

To send an email, hit Save AS > Alert and configure it run on a schedule and to trigger an email action when number of results is greater than 0 (and list in triggered alerts for verification/troubleshooting)

alt text

Also note @hexx answer that an alert for overall usage exists in the management console and can be triggered as an email alert as well.

- MattyMo

a212830
Champion

Thanks, that worked for the evaluation. I'd like to be able to trigger a separate search through the alert - is that possible?

0 Karma

mattymo
Splunk Employee
Splunk Employee

yep, the alert action framework or the old run a script methods allow you to do what you want at that time. What do would you want to search if this alert hit?

- MattyMo
0 Karma

sloshburch
Splunk Employee
Splunk Employee

As in, make the search trigger another search?

0 Karma

a212830
Champion

Yes. This alert will indicate a problem, and the customer is going to want some data to do research, so I want to then run a report giving him the needed data. Actually, something missing from the product - should be able to request an action that just points to another saved search and run it as an action. (At least, IMHO...)

0 Karma

sloshburch
Splunk Employee
Splunk Employee

A couple of options here:

  • Have you validated that no such thing already exists on splunkbase? The new mod-alert framework is designed for such extensibility
  • Pre mod-alert, we used to solve this by include a dynamic URL field in the results. That way when the alert is triggered, one of the fields that would be sent to the customer (assuming you include the results in the alert) is the URL that will direct them to learn more
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...