Splunk Search

Limits that shouldn't be exeeded and potential big data

iKate
Builder

Hi everyone,
below are several questions and each of them is very important for us. Hope for your help.

As written in docs these two limits have maximum values that shoudn't be exeeded:

[searchresults]
maxresultrows = <integer>
* Configures the maximum number of events are generated by search commands which 
* This limit should not exceed **50000**. Setting this limit higher than 50000 causes instability.
[subsearch]
* This stanza controls subsearch results.
maxout = <integer>
* Maximum number of results to return from a subsearch.
* This value cannot be greater than or equal to **10500**.
* Defaults to 100.

Is it the same for any hardware configurations or at what hardware capacity its increasing will not harm the system?
At what extent can they be risen?

At the moment our standalone server has the following configuration:

Intel x86-64-bit chip architecture 
Standard Linux or Windows 64-bit distribution 
2 CPU, 4 core per CPU, 2.5-3Ghz per core 
8GB RAM 
4x300GB SAS hard disks at 10,000 rpm each in RAID 10 
capable of 800 IO operations / second (IOPS) 
standard 1Gb Ethernet NIC, optional 2nd NIC for a management network

and local/limits.conf file is set to

[searchresults]
maxresultrows = 300000
[subsearch]
maxout = 1000000

default/limits.conf has 50000 and 10000 respectively.

But when e.g. joining results they are still truncated if there are more than 50000 results..

Why local settings are not accepted?
And I also wonder why is there such low default limit on joining at all?

And another thing:
What specifics in limits configuration should have such heavy companies like eBay?

lguinn2
Legend

You can't set maxout to 1 million. It must be less than 10,500.

However, individual commands (like join) may have their own limits. The default limit for join is actually 50,000. I don't know if you can set it any higher.

However uf you set maxresultrows greater than 50,000 - limits.conf clearly states that you risk de-stabilizing your system.

Splunk can search many millions of events - these limits apply only to how many results will be returned from a search or subsearch. What do you plan to do with so many results? If you share a particular problem with the community, we might be able to help you identify a more efficient approach.

Finally, I don't know (or care) about the implementation details, but I expect that these limits are set to prevent searches that would overwhelm the server's resources such as memory and temporary disk space. Just my guess.

srioux
Communicator

Our experience has been that Splunk grows well horizontally - if you need to do more with it, then add more servers to the mix (search head or indexers, or a seperate stack). Hardware limitations are more likely to cause it to choke than Splunk's actual defined limits.conf options (ex: IOPS, system limits for open file descriptors, too many simultaneous searches, etc).

As lguinn denotes, there are reasons why the default limits are in place. If you can provide context on what you're looking to do with 50,000+ rows, there may be alternate options (searches against summarized results, etc).

0 Karma

lguinn2
Legend

AFAIK, large customers with successful Splunk implementations follow the server sizing guidelines in the documentation. When they want to index more data, they add more indexers - they don't up the limits. I know of a number of customers who are indexing terrabytes of data per day.

The Distributed Deployment Manual has the sizing guidelines.

So: the answer is (1) add more indexers of the recommended size - don't increase the hardware size of the indexers - and (2) don't exceed the maximums.

0 Karma

iKate
Builder

thanks for the response! but all my questions are still opened)

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...