What are the proper user quotas to protect our indexers?

[danielbb]

Yesterday, one indexer got crashed due to a very badly developed dashboard - it instantly consumed all the memory of the indexer.

Which quotas should we place in order to prevent such cases?

Btw, it does seem that this particular indexer ran all or most of the queries of this dashboard, which is weird.

[bandit]

A few things you can look at:
You can modify times.conf and verify users are not searching All time by default
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Timesconf
https://answers.splunk.com/answers/79547/disable-all-time.html

Force users to specify indexes in their queries by verifying that "Indexes searched by default" for the role assigned to the users is NOT set to "All non-internal indexes"

Verify that data is being distributed equally to indexers (more or less) otherwise the workload won't be evenly distributed and you'll be waiting longer for the query to complete if one indexer is doing most of the work. splunk_server is the field name representing the indexer that is hosting/serving the data.

your base query | top 100 splunk_server

[somesoni2]

Have a look at this:
https://docs.splunk.com/Documentation/Splunk/7.3.1/Search/Limitsearchprocessmemoryusage

[danielbb]

That's great. Btw, what's the default max memory allocation per a search query?

[danielbb]

@somesoni2 - I see these values on the indexers -

$SPLUNK_HOME/etc/system/default/limits.conf - enable_memory_tracker = false
$SPLUNK_HOME/etc/system/default/limits.conf - search_process_memory_usage_percentage_threshold = 25
$SPLUNK_HOME/etc/system/default/limits.conf - search_process_memory_usage_threshold = 4000

Should we change these values on the indexers and on the SHs as well?