Alerting

Trigger is firing when it shouldn't

icodebro
New Member

Hi

I have the following alert setup

alt text

With custom time range:

alt text
So looking at results in the last 2 hours.

I expect to receive an alert if there are zero results in the last 2 hours from 8am to 4pm, Monday to Friday.

However I am receiving an email at every interval (8am, 10am, 12pm, 2pm, 4pm), Monday to Friday.
And of course, I have manually run the query for the exact time frame which yields results.

Not sure where I have gone wrong...so I'd appreciate if anyone can help!

Many thanks

Tags (2)
0 Karma

Vijeta
Influencer

@icodebro - You can check in your Activity->Job , search your job name and in Actions , click on Job, Inspect job and you can see the count of results and search log. Also you can click on the job and see the results for last 2 hrs.

0 Karma

icodebro
New Member

The job inspector does indicate no results.
I then opened the search query from the job list, which also says no results found.
Now on that same page, without modifying any part of the query, I click search (magnifying glass), results are returned.

Any idea why I get different results?

Logs are ingested in real time because I get results even before the alert if fired.

0 Karma

maciep
Champion

can you share your search, masking anything sensitive? Some thoughts....

Have you looked in the _internal or _audit index for this saved search to confirm that it did in fact return 0 results before firing the alert?

Is this data being ingested in real-time? Meaning, for that 2 hour period, is the data definitely in Splunk at that time? Or is maybe ingested every 3 hours or something like that?

Do you have any knowledge objects (field extraction, lookup, etc.) used in your search that may not be available to the search due to permissions or which app they're in, etc.?

0 Karma

to4kawa
Ultra Champion
0 8,12,14,16 * * 1-5    Every 8,12,14,16 hours, Monday through Friday.

It works according to your settings.
in the last 2 hours from 8am to 4pm, Monday to Friday.
I'm not sure what you say.
This is between 2pm to 4pm?

Alert/CronExpressions

0 Karma

icodebro
New Member

Thanks for responding.

What I meant was that I expect to receive an alert if there are zero results in the 2 hour window when the cron expression is matched. eg. if there are zero results returned from 10am-12pm, Monday.

0 Karma

to4kawa
Ultra Champion

when the cron expression is searching time.
when the cron expression is matched
you should do with another way.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...