Alerting

Why the alert did not trigger for below cron expression?

abhi04
Communicator

16-59/10 5-6 * * * cron was setup for more than 0 events.

We had an event at 5:15 Am. Any idea why the alert did not trigger?

The query used is for -5m@m

Tags (1)
0 Karma

woodcock
Esteemed Legend

Just because your event happened at that time does not mean that it was indexed and searchable at the time the search ran. A window so short as "within the last 5 minutes" leaves very little time for pipeline latencies which are common forwarding events into Splunk. If you compare the value of _time with _indextime for that event and they are more than 5-minutes apart (300 seconds), then the latency indicates that the event was not searchable in Splunk when the search looking for it ran.

0 Karma

woodcock
Esteemed Legend

And before @mattymo says, it: Meta W00t!

0 Karma

FrankVl
Ultra Champion

With that cron schedule, I guess the search ran first time at 5:20 AM? Did you confirm the search actually ran, and indeed ran at that time?

0 Karma

abhi04
Communicator

@FrankVI
Should not the search run at 5:16 and check for last 5 minutes? Also, how to check when the search ran at that time?

0 Karma

abhi04
Communicator

I just checked and confirmed that the it is scheduled 05:16:00

0 Karma

FrankVl
Ultra Champion

Hmm, I might be wrong about that then. I also checked with crontab guru and that agrees with you that it would run at 16,26,36,46,56 : https://crontab.guru/#16-59/10_5-6_*_*_*

Note: I added 2 stars at the end to make it a proper complete cron schedule.

From the settings page for saved searches, you should see a "View Recent" link in the actions column. Which allows you to inspect recent search executions. Also saved search executions are logged in index=_audit.

0 Karma

FrankVl
Ultra Champion

No, you set it to /10, so it runs at 0,10,20,30,40,50 (where 0 and 10 are skipped because of your 16-59 time window).

0 Karma

abhi04
Communicator

According to me,cron expression = 16-59/10 5-6 * * * means the search query will trigger at 5 hours and between 16 to 59 minutes in a span of 10 minutes, same for the hour 6.

So it will run,

5:16, 5:26, 5:36, 5:46, 5:56 and same for 6th hour

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...