Alerting

How to NOT alert on events that are caused by a reboot

cmahan
Path Finder

What is the best way to not alert on events that are caused by a reboot. For example - I am alerting on a WMI:Service Status=Stopped for a particular service. I don't want to get an alert if the server was rebooting and that is why the service was in the stopped state. Might be a bad example as the result might not be what I expect, but I need to be able to accomplish the general concept for other alerts. I think I am looking for a subsearch? but not sure how to write it..?

blah blah event | subsearch NOT "i just rebooted" within last 10 minutes

something along those lines

Then if the event or state is detected, but also a reboot event was detected within the last 10 minutes, then no alert is triggered. If there was no reboot, the alert IS triggered.

Tags (2)
0 Karma

MuS
SplunkTrust
SplunkTrust

Hi cmahan,

take a look at this run everywhere command:

index=_internal sourcetype=splunkd series=splunkd earliest=-11min@min 
| bucket _time span=10min 
| search NOT [ search index=_audit action=search ] 
| timechart span=1min count

it searches for events in the _internal index over the last 11 minutes, creates buckets of 10 minutes span and searches within this time span if there was no match for action=search in index=_audit. Yes this is a non-sense use case but it will help you understand your alert 😉

Based on this and your comments, try something like:

sourcetype="WMI:Service" Name=slinksc State=Stopped earliest=-11min@min 
| bucket _time span=10min 
| search NOT [ search  EventCode=1074 ]

I removed the map command, because it is just too messy.

cheers, MuS

cmahan
Path Finder

Thanks! I will give that a try.

0 Karma

cmahan
Path Finder

No dice. Tried both of these without success.

sourcetype="WMI:Service" Name=slinksc State=Stopped | search NOT [ search earliest>=-10min@min EventCode=1074 ]

sourcetype="WMI:Service" Name=slinksc State=Stopped | eval time=_time | map search="search NOT EventCode=1074 earliest>=($time$ - 600 ) latest<=($time$)"
0 Karma

MuS
SplunkTrust
SplunkTrust

okay, twice my bad....

just learned this answer won't work for the first one using subsearch!! Why? Because Splunk runs the inner search using the same info from the timerange picker as the outer search.
And the map is also wrong, because the NOT will not work 😞 ... I'll work on a updated version....

0 Karma

MuS
SplunkTrust
SplunkTrust

update ping, see the updated answer

0 Karma

cmahan
Path Finder

Tried it with great hopes! but alas, the event still shows up. It seems like its ignoring the subsearch still. Are we sure that syntax is conveying "don't return anything if you find results matching the subsearch along with the main search, even if the main search has results?" I'm actually trying with a different service now. Here is what i have tried - only difference between the 2 is the way I defined the State to include all states besides Running:

sourcetype="WMI:Service" Name="ds_agent" State="Stopped" OR State="Start*" earliest=-11min@min | bucket _time span=10min | search NOT [ search EventCode=1074 ]

and

sourcetype="WMI:Service" Name="ds_agent" NOT State=Running earliest=-11min@min | bucket _time span=10min | search NOT [ search EventCode=1074 ]

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...