I have ran into same issue. Started with Splunk 6.5 as deployment server on a VM with 6 CPU and 16GB ram. After restarting splunk service Splunk Universal forwarder management is very responsive. As it builds client list and number of clients climb over 1000 you can notice considerable degradation of response from splunk web. (Only under Forwarder Management section all other are fine). I have 6200 Clients and by the time Splunk Deployment builds complete list, the Web UI response time goes into minutes (3 minutes typically).
So I upgrade from VM to a physical box with 16 CPU and 48GB ram. Turned THP off and set my ulimits as follows to remove any bottle necks.
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63621
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16284
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Installed Splunk 7.1.1
My Cpu utilization hovers between .05% to .15%. Except when I try to reload the Universal management page at which I see one Cpu getting to 100% while others are less than a percent. The performance gains I have seen after moving to Physical server
are VM Page load time just over 3 minutes, Physical Server page load time Just over 1 Minute. It seems Every 1000 Machines add 10 Seconds delay.
I Verified the poor response in Chrome, Firefox, IE, Edge, Safari. Which proved that is is not Browser related. Then I used firefox debug to different sections of page and their load time. See the image. It is clearly a bug in Splunk which needs immediate attention.
For some reason people could not open the image. Please use the following link.
http://tinypic.com/r/o9lu84/9
... View more