Alerting

DM missing forwarders (ALL)

splunkreal
Motivator

Hello guys,

since 6.5.2 I regularly receive this kind of (fake) alert :

The alert condition for 'DM missing forwarders' was triggered.
Forwarder,Source IP,Forwarder Type,Last Connected,Current Status
(ALL),(ALL),Splunk forwarder,1493074800,missing

Thanks for your help!

* If this helps, please upvote or accept solution 🙂 *
0 Karma

dineshraj9
Builder

You can disable the alert "DMC missing forwarders" under "splunk_monitoring_console" app.

However, it is a useful search which lists the forwarders which are not reporting to your environment any longer. You can optimize the search to show only those servers which have not responded for more than an hour, in last 24 hours by modifying the search to -

| inputlookup dmc_forwarder_assets
| search status="missing" 
| rename hostname as Instance | eval diff=round(now()-last_connected) | where diff>3600 AND diff<86400 | convert ctime(last_connected) 

This way the list will be small and you can restart the forwarder instances which are down.

splunkreal
Motivator

does it mean this is a legitimate alert? all forwarders are disconnected at once?

* If this helps, please upvote or accept solution 🙂 *
0 Karma

dineshraj9
Builder

Is your DMC connected to all indexers that are receiving data from the forwarders?
In our environment we have our custom search to monitor such forwarder issues.

| metadata type=hosts | eval diff=now()-lastTime | where diff > 3600 AND diff < 86400 | convert ctime(lastTime) | table host lastTime

Check if the results match.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi realsplunk,
if you go in http://your_server:8000/en-US/app/splunk_monitoring_console/alerts
you can find all the alerts of Distributed Monitoring Console.
Go in "DMC Alert - Missing forwarders" Alert and disable it.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...