Why the Splunk down and how should I fix it?

[urapaveerapan]

Hi,
When many users access the Splunk at the same time or even I test by open several dashboards at a time, the Splunk is occasionally not working and the application show “This page can’t be displayed”
As a workaround, I must restart Splunk to make it back to normal.

There are no issues on the VM server performance.

Our current system is One instance VM server located in our office building and access by Intranet
Linux server 3.12.49
Memory : 12 GB
CPUs : 12 vCPUs
Disk : 500 GB
Incoming data < 2 GB/day

Note that, it usually have message "Maximum concurrent search..", is it the reason that can make Splunk down?
Does anyone have any ideas?

[gcusello]

Hi urapaveerapan,
surely it's a performance issue, at first you should check the disks IO that Splunk recommends must be at least 800 IOPS (better 1200): there are some open source tools like Bonnie++ to do that.
Anyway there are some dashboards in Splunk Monitoring Console that help you to understand if there are queues in indexing or in searching.
Remember that every search (if in a dashboard you have 10 panels, there are 10 running searches) takes and uses a CPU, so if you have more than 12 searches in the same time there is a search queue.
If in addition you're using many real time searches you overload your system.

The solution to your problem is to analyze your requirements in terms of users and how much they use system (searches, panels, indexing, ...), so you can design your architecture: maybe you need more indexers or to use a distributed architecture and/or maybe you need to redesign your dashboards:
I had a customer with some dashboards with 10 real time panels used at the same time by many users, solution was to add more indexers and replace real time serches with scheduled reports.

Bye.
Giuseppe