Realtime searches in ES delayed?

[abonuccelli_spl]

Hi,

I've installed Enterprise Security dedicated search head following all the best practices with beefy enough hardware specs and when running a simple realtime search, compared to a Splunk standard search head I'm getting a 60s+ delay before I get the same result.

What is happening?

[abonuccelli_spl · Accepted solution]

ES app is using by default Indexed real-time_search

To lessen the impact on the indexer, you can enable indexed real-time search. This will basically run the search like a historical search, but will also continually update it with new events as they appear on disk.

/opt/SPLUNK/6.0.3-ES-SH/splunk/bin $ ./splunk btool limits list realtime --debug
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   [realtime]
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/default/limits.conf alerting_period_ms = 3000
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   blocking = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   default_backfill = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   disabled = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   enforce_time_order = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_cluster_update_interval = 30
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_default_span = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_disk_sync_delay = 60
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_maximum_span = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   indexed_realtime_use_by_default = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   indexfilter = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             max_blocking_secs = 60
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             queue_size = 10000

This is a necessary compromise between having the result in a matter of seconds vs 1+ minute and having lots of 'realtime' searches able to run concurrently.

To shorten the delay, you can attempt to decrease paramater

/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_disk_sync_delay = 60

However this shouldn't be set to a value lower than 30 (seconds).

If in doubt, please contact Splunk Technical Support.

[ehudb]

One of my customers is building its ES environment, and have real trouble with that realtime delay.
What is the meaning of reducing "indexed_realtime_disk_sync_delay" to the minimum (for example 1)

Why should it be at least 30?

Thanks.