Alerting

Setting up an alert criteria for sourcetype counts

asarolkar
Builder

I have a basic search like this which counts ALL sourcetypes logged for a certain index->

index="syslog" sourcetype="alpha" OR sourcetype="beta" OR sourcetype="gamma"  | stats count as totalRuns | search totalRuns < 4

We know that alpha, beta and gamma are the three types of sourcetypes that can be logged against this index. I set the cutoff to 4, because one of these actually gets logged twice.

In the "Create an alert" section, I set the search ABOVE as the search criteria and the FILTER as follows ->

[ If the Number of Events ] is [NOT EQUAL TO] 4, this alert really should be raised whenever i get less than 4 counts.

This alert gets triggered at a certain cron job time.

However, EVEN IF there are 4 events, it gets triggered.
My problem is that that shouldn't happen. How do I tailor my alert so that this doesn't happen ?

I have tried all other permutations, where I set the filter to [ If the Number of Results ] is [GREATHER THAN] 0.

And that has not helped either.

Tags (2)
0 Karma

yannK
Splunk Employee
Splunk Employee

Try to add the condition in the search with a where.

index="syslog" sourcetype="alpha" OR sourcetype="beta" OR sourcetype="gamma" | stats count as totalRuns | where totalRuns < 4

the setup the alert to trigger every time the number of results is > 0

0 Karma

asarolkar
Builder

Do you know what the filter criteria for that ought to be ?

0 Karma

yannK
Splunk Employee
Splunk Employee

my bad, if you really want to detected a missing sourcetype, your search has to be a distinct count of the sourcetypes.

index="syslog" sourcetype="alpha" OR sourcetype="beta" OR sourcetype="gamma" | stats dc(sourcetype) AS number_sourcetype | where number_sourcetype < 4

0 Karma

asarolkar
Builder

Alert was triggered because of: 'Saved Search [AlertSourcetypeFailure]: custom(1)'

0 Karma

asarolkar
Builder

That did not work 😞

0 Karma

jonuwz
Influencer

The search in your saved search should be :

index="syslog" sourcetype="alpha" OR sourcetype="beta" OR sourcetype="gamma"  | stats count as totalRun

And you should set a custom condition for the alert defined as :

search totalRuns != 4
0 Karma

jonuwz
Influencer

And what happens if you look at the results for that search ? Is totalRuns = 4 ?

0 Karma

asarolkar
Builder

Alert was triggered because of: 'Saved Search [AlertSourcetypeFailure]: custom(1)'

0 Karma

asarolkar
Builder

yes. it keeps on firing the stupid thing every time.

0 Karma

jonuwz
Influencer

What happened ? does it fire every time or not at all ?

0 Karma

asarolkar
Builder

Hi, I tried that and unfortunately, that did not work out.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...