Alerting

How to schedule an alert to run at 11:00PM PST (23:00) every day and trigger an email if the number of events is less than 15?

mprreddy51
Explorer

Hi All,

Here is my requirement:

I want to create a alert if the number of events is less than 15. It should trigger email everyday at 23.00 (11.00pm pst)

Below is the search:

Index=abc sourcetype=pqr "ERROR"

The cron job I used is 00 23 star star star ( 00 23 * * *)

starttime: -24h
finishtime: now

Here is the actual problem:
If I schedule it for every 5 minutes or 10 minutes, the email is triggering and working fine (testing purpose). Once if I change for 23.00hrs(11.00 pm pst), the alert is not working and not triggering an email and there are no other searches running at that time.

Please help.

Below is my savedsearches.conf:

[ERROR]
action.email = 1
action.email.inline = 1
action.email.sendresults = 1
action.email.to = abc@gmail.com
alert.digest_mode = True
alert.suppress = 0
alert.track = 1
auto_summarize.dispatch.earliest_time = -1d@h
counttype = number of events
cron_schedule = 00 23***
dispatch.earliest_time = -24h
dispatch.latest_time = now
enableSched = 1
quantity = 15
relation = less than
schedule_window = 5
search = index=abc sourcetype=pqr "ERROR"

alt text

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Well, see the search head utilization and you can either increase resources OR add more SH accordingly. For this issue, I guess all these searches are running with default splunk-system-user accound (means owner=nobody), and it's hitting it's limit big time. You should change the owner to specific user accounts needing this data, preferably power user/admin type of account so that their job quota is used to run these searches instead of default splunk user.

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Well, see the search head utilization and you can either increase resources OR add more SH accordingly. For this issue, I guess all these searches are running with default splunk-system-user accound (means owner=nobody), and it's hitting it's limit big time. You should change the owner to specific user accounts needing this data, preferably power user/admin type of account so that their job quota is used to run these searches instead of default splunk user.

0 Karma

mprreddy51
Explorer

Thanks somesoni,

It is working now after changing the owner to admin.Thanks

0 Karma

somesoni2
SplunkTrust
SplunkTrust

If you run this search manually, how many events do you get?? Also, when you run every 5 or 10 minutes, do you keep the same start time and finish time??

Index=abc sourcetype=pqr "ERROR" earliest=-24h
0 Karma

mprreddy51
Explorer

Hi somesoni,
Yes, If i schedule a cron for every 5 min or 10 min , i am using the earliest time(-24h) and finishtime(now).
and it should trigger email irrespective of the number of events. because my condition is less than.If there is no events also it should trigger email.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Agree.. Can you try to set the value for "Schedule Window" to 0 (as per screenshot it's 5)?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Also, can you check if the your saved search was executed and didn't trigger alert?

index=_internal sourcetype=scheduler savedsearch_name="Error"
0 Karma

mprreddy51
Explorer

Hi somesoni,

Yes i made window 0 and i checked .but same result.If i check splunkd it is showing below error

"02-18-2016 12:10:52.760 -0800 WARN SHPMaster - Search not executed: Your maximum number of concurrent searches has been reached. usage=2828 quota=150 user=splunk-system-user. for search: nobody"

For this i increased srchjobsquota also no change.should we increase any crores?
Thanks

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...