Hi,
I fought with the same issue and solved it. Same setup. 1 Splunk Server with 1 Splunk Forwarder. The issue is the way the saved searches are configured in the Splunk App for Unix and Linux. Below is the relevant stanza in its default configuration.
[Processes_Exceeds_by_Host]
action.summary_index = 1
action.summary_index.marker = unix_aggregated_alerts
action.summary_index._name = unix_summary
alert.digest_mode = True
alert.expires = 1d
alert.suppress = 1
alert.suppress.period = 1m
alert.track = 1
auto_summarize.dispatch.earliest_time = -5m@m
cron_schedule = */5 * * * *
counttype = number of events
disabled = 1
dispatch.earliest_time = -5m@m
dispatch.latest_time = now
displayview = unix_flashtimeline
enableSched = 1
quantity = 1
relation = greater than
search = `Processes_Exceeds_by_Host("`_unix_alert_threshold_Processes_Exceeds_by_Host`")`
The issues is the combination of the quantity and relation setting. In plain english it's saying "if # of results returned is greater than 1 fire the alert". When you have ONE server it will never fire. Also, it would hide the fact that there is a server firing alerts.
The fix:
copy $SPLUNK_HOME/etc/apps/splunk_app_for_nix/default/savedsearches.conf to $SPLUNK_HOME/etc/apps/splunk_app_for_nix/local/savedsearches.conf
Edit $SPLUNK_HOME/etc/apps/splunk_app_for_nix/local/savedsearches.conf
Change quantity under [Processes_Exceeds_by_Host] to 0
I know this is an old question, hopefully it helps someone else.
-Felix
... View more