Our current Splunk deployment is around 300 servers. We have all of those systems in our DMC and our able to get data from them successfully. However, when running the Health Checks it only ever checks 100 systems, not the full spectrum of our systems. Is there a way to change the Health Checks so that they'll work on any number of Splunk instances that meet the DMC group requirements instead of limiting it to 100? The primary check I've been working with is the Assessment of Server ulimits since that checks all servers globally.
I've checked through some of the searches in the health checks and can't find anything there that specifically limits them to 100.
We're currently running 6.6.1 on the DMC and the other Splunk instances.
This is currently not supported. An enhancement request has been submitted to Support.
This is currently not supported. An enhancement request has been submitted to Support.
The searches for the health check are visible in the job inspector.
Look and see if the searches are returning all of the data... Maybe it is simply a display issue.
I checked the job inspector and I see that it definitely looks like it's hitting all of the servers - I see the remoteSearchLogs for all of the appropriate systems and invocation counts look accurate and expected. The resultCount in the inspector shows 235 (which is pretty accurate) - it just only shows results of 100 in the display.
I would
1) Use the job inspector to make sure your systems are OK. It isn't ideal, but it works
OR
2) Use instance grouping to run the health check on groups that are less than 100
Also, if this is a feature you want to see ( or you consider it a bug ) either way, reach out to support. They can make sure your voice is heard.