On the incident posture screen the Informational --> Critical boxes update and show the proper number and status of events.
The Recent Incidents do not show all triggered alerts.
App installed in distributed environment using non-standard index search head is stand alone, the index is sent to clustered indexers, summary indexes are store on the search head.
When an event triggers I am seeing the following logs:
2015-04-09 11:50:06,187 DEBUG Create event will be: time=2015-04-09T11:50:06.187492 severity=INFO origin="alert_handler" event_id="f5e728746d9ec28b42db2b41ba85109e" user="splunk-system-user" action="create" alert="XXXX Alert Name" incident_id="624b2d98-14df-43b9-9765-fac36e8662e0" job_id="scheduler_mesearch_RMD5853430e3bafc3e3f_at_1428605400_164" result_id="0" owner="unassigned" status="new" urgency="high" ttl="86400" alert_time="1428605401"
action = create
eventtype = failed_login eventtype = nix-all-logs eventtype = nix_errors error
host = Host.name
index = _internal
source = /opt/splunk/var/log/splunk/alert_manager.log
sourcetype = key_indicators_controller-2
When checking the Recent Incident screen or by searching | all_alerts I do not see the alert listed.
The counters for the informational --> Critical count up but there isn't an incident to respond to. It appears almost random, originally I thought it was being truncated to I increased the Truncate in props.conf to allow for larger than 10000, but that hasn't fixed the issue.
Any ideas on what could cause this?
... View more