Getting Data In

What are the proper user quotas to protect our indexers?

danielbb
Motivator

Yesterday, one indexer got crashed due to a very badly developed dashboard - it instantly consumed all the memory of the indexer.

Which quotas should we place in order to prevent such cases?

Btw, it does seem that this particular indexer ran all or most of the queries of this dashboard, which is weird.

alt text

Tags (2)
0 Karma

bandit
Motivator

A few things you can look at:
You can modify times.conf and verify users are not searching All time by default
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Timesconf
https://answers.splunk.com/answers/79547/disable-all-time.html

Force users to specify indexes in their queries by verifying that "Indexes searched by default" for the role assigned to the users is NOT set to "All non-internal indexes"

Verify that data is being distributed equally to indexers (more or less) otherwise the workload won't be evenly distributed and you'll be waiting longer for the query to complete if one indexer is doing most of the work. splunk_server is the field name representing the indexer that is hosting/serving the data.

your base query | top 100 splunk_server

somesoni2
SplunkTrust
SplunkTrust

danielbb
Motivator

That's great. Btw, what's the default max memory allocation per a search query?

0 Karma

danielbb
Motivator

@somesoni2 - I see these values on the indexers -

$SPLUNK_HOME/etc/system/default/limits.conf - enable_memory_tracker = false
$SPLUNK_HOME/etc/system/default/limits.conf - search_process_memory_usage_percentage_threshold = 25
$SPLUNK_HOME/etc/system/default/limits.conf - search_process_memory_usage_threshold = 4000

Should we change these values on the indexers and on the SHs as well?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...