Splunk ES Duplicate of Notable Events displaying in Incident Review Dashboard

[dsofoulis]

Hi Everyone,

I have an issue where I am seeing am seeing duplicate notable events for a single event.
So heres the details:
-First off, this is not occurring for every notable, its random.
-cron schedule is every 20 minutes. With a time range of -60min
-There is only one trigger event, and only one event in the notable index.
-The duplicate and the original appear at the same time.

I've also had other issues where depending on what urgency I've filtered out, can impact the values of the other urgencies. For example I have 50 highs and 100 lows, I now filter out the lows and high drops to 25. I'm thinking this randomness may be related to my duplicate notable event issue.

ES version 5.2.0
Splunk Enterprise version 7.1.3.1

[starcher]

The below is from Ovie in the Splunk community Slack channel #enterprise-security: Not sure if it is your issue but it is supposed to be related to duplicates like that. So it might apply to some of you.

This can happen because of phased_execution_mode

https://docs.splunk.com/Documentation/ES/5.2.0/RN/KnownIssues
2019-04-08 SOLNESS-18603 Incident Review: eventCount does not match resultCount causing display issues (such as events being displayed twice)

Workaround:
Set phased_execution_mode to singlethreaded

For: limits.conf

[search]
phased_execution_mode = singlethreaded

https://docs.splunk.com/Documentation/ES/5.1.0/Install/DeploymentPlanning#Splunk_Enterprise_system_r...

Splunk Enterprise Security 5.1 is compatible with Splunk Enterprise 7.1.0 and 7.1.1 only by setting phased_execution_mode=singlethreaded in the [search] stanza of the $SPLUNK_HOME/etc/system/local/limits.conf file to avoid an issue that is fixed in Splunk Enterprise 7.1.2. However, if you apply this workaround for 7.1.0 and 7.1.1 and then upgrade Splunk Enterprise but remain on ES 5.1, then you need to set it back to phased_execution_mode=multithreaded.

Splunk Enterprise Security 5.2.x is compatible with Splunk Enterprise 7.1.0 and 7.1.1 only by setting phased_execution_mode=singlethreaded in the [search] stanza of the $SPLUNK_HOME/etc/system/local/limits.conf file to avoid an issue that is fixed in Splunk Enterprise 7.1.2.

Bottom line is this setting has caused some serious grief.

[sryedudo]

Any resolution yet ? We are running into same issue. (Splunk 7.1.2, ES 5.1.0)

[tfellinger]

We did not had this problem for a while, but now it came back. I will open a case and see what the splunks say about it.

[shrutheen]

Same issue here. Just upgraded the ES Search head cluster to 7.1.3.1. However, the ES App version is 5.1.1and indexers are running with a lower version.

[tfellinger]

Same here with 7.2.1 and ES 5.2.0. Anyone fould a solution yet?

[shrutheen]

I was told by Splunk support that i was hitting SPL-160881 which was been partially resolved in 7.1.2 and later as well as ES 5.2.1 or later. However, we upgraded our setup to 7.1.6 and ES to 5.2.2 post which the duplicate occurrences have been very rare.
As you know the issue is only with the backend code of incident review dashboard which displays duplicate notables, although they do not actually exist. Splunk support had made it clear even before we upgraded setup to 7.1.6 and ES to 5.2.2, that the issue was not completely fixed yet and may recur rarely.

[tfellinger]

Thanks shrutheen! I will try ES 5.2.2 and see if that helps!
Cheers, Thomas