Enterprise Security (ES) asset lookups failing with SRC and dest fields reporting “unknown-internal” and “unknown-external”

[ekost]

Running ES 5.1 on Splunk 7.1. The asset lookups have been working fine. This morning the SRC and dest fields display “unknown-internal” and “unknown-external” in place of the usual asset information. The lookups are populating normally, and nothing’s changed on the ES Search Head recently.

Digging deeper, the behavior makes it appear that the CIDR lookup asset_by_cidr.csv is happening before the string lookup assets_by_str.csv. But why would that be, and what would change the normal order-of-operations behavior for those lookups?

[ekost]

The size of the assets_by_str.csv lookup is exceeding the limits.conf setting max_memtable_bytes, and is being treated as a batched lookup. Splunk will manage lookups in-memory for CSVs less than 10MB (by default,) and index the rest as external batch based. The in-memory lookups get executed immediately, and order is consistent. For batched lookups, Splunk waits until a certain batch size is reached before performing a lookup operation. Due to this, there are higher chances of running into out-of-order conflicts when utilizing indexed lookups.

Raise the limits.conf setting max_memtable_bytes to a value larger than your assets_by_string.csv lookup on the SH and Indexers to eliminate the issue. Note: this will use more RAM on the hosts.