Alerting

Alert fired but I don't know why

riotto
Path Finder

I had an alert that fired which shows a condition that the indexer hadn't received a specific kind of event within the last 5 minutes, but
it had received it. I looked at the _indextime of these events and it shows that they were indeed indexed within those 5 minutes. Is there a log that I can look at that might show if the indexer was doing some kind of housekeeping and the events weren't technically
indexed yet?

Tags (1)
0 Karma

riotto
Path Finder

yes...I posted that I looked at _indextime

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi riotto,
did you verified when you received these events using _indextime?
you can run a search like this

index=your_index
| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S"), alert_time=strftime(now(),"%Y-%m-%d %H:%M:%S")
| table _time indextime alert_time

maybe you received them after the alert running.

Bye.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

yes but event timestamp (not _indextime) is in the time range of your search?
there are two choices:

  • events arrived after the alert time period (you can exclude this situation comparing indextime with the alert_time);
  • events arrived before the alert period but with a timestamp outside of the above time period (you can exclude this situation comparing _time with the alert_time)

Bye.
Giuseppe

0 Karma

riotto
Path Finder

The alert triggered at 21:08. This alert runs every 2 minutes and looks at the last 5 minutes of indexed events. It counts how many of these specific events were indexed in that last 5 minutes. it counted 0 events. BUT, if I look at that time period 21:03 - 21:08 of when the alert counted 0 events and examine the _time and _indextime of those events, it shows 100s of these events with an _indextime of milliseconds of the _time fpr each of them. Am I looking at this wrong?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...