Splunk Search

In real-time alert, if I use lookup command, too many alerts triggered.

yutaka1005
Builder

Splunk ver 7.1.1

I'm using real-time alert that trigger when there is event which has src_ip match black_list.csv like below.

index=hogehoge 
| lookup black_list.csv src OUTPUT status
| where isnotnull(status) 
| table _time src status

But if there is such a event, this alert is continuously triggered per 5 seconds against same event!
I think it caused by lookup command touches every event again, when new event is arrived.

Should not I use lookup in real-time alert?
And is it specification?
* If this is written in the document, I'm sorry ....

Please someone help me.

0 Karma

DalJeanis
Legend

You probably don't need a realtime alert for this. Realtime should be reserved for ultracritical items where the SLA is seconds, rather than hours or minutes.

First, it's better to schedule a periodic scheduled search that runs every 1m, 2m, 3m or even 5m depending on what the real SLA for response is.

Second, find out what your typical delay is for indexing, and give your system at least that many minutes to process before your scan. Most installations ingest events in less than a minute, so if you're running every 3 minutes (say at 2,5,8....56,59 minutes after the hour) Then you might set the search up as

earliest=-5m@m latest=-2m@m

Third, for this particular alert, it seems to be the src that you want to alert on, so you should put a |dedup src or stats count by src so that there is only one record coming out.

Fourth, you probably want to throttle the results so that, once an alert has been received for one src, you suppress further alerts until some length of time has passed.

So, that scheduled alert would look something like this..

 index=hogehoge earliest=-5m@m latest=-2m@m
 | dedup src
 | lookup black_list.csv src OUTPUT status
 | where isnotnull(status) 
 | table _time src status

You will want throttle -> per result -> src

https://docs.splunk.com/Documentation/Splunk/7.1.2/Alert/ThrottleAlerts

0 Karma

jkat54
SplunkTrust
SplunkTrust

This would trigger an alert every time something matches the lookup which is presumably 100% of the time.

what are the values of the” status” field? If they are “up/down”, then you could do this.

| where status==“down”

Instead of the where clause you have currently.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...