Is there a way to configure new deployment server a new server only for the purpose of reports and alerts?

[pdantuuri0411]

We have 1 deployment server where data is being forwarded from around 60 forwarders. Lately, we are noticing that the Splunk server is not performing to the full potential due to the load it is taking. We have a lot of reports and alerts configured, which is taking a large amount of CPU usage.

So we are planning to configure another deployment server. Is there a way to configure a new server only for the purpose of reports and alerts?

I am new to Splunk — please advice.

[woodcock]

Here is what I would do, in the order that I would do it.
Upgrade to the latest v7.2.
Install this app:
https://splunkbase.splunk.com/app/4409/

In indexes.conf on each indexer:

#https://docs.splunk.com/Documentation/Splunk/7.2.4/Installation/AboutupgradingREADTHISFIRST#Important_upgrade_information_and_changes
[default]
tsidxWritingLevel=2

In the deploymentclient.conf in the deployment-apps directory on your Search Head (and push it out):

[deployment-client]
phoneHomeIntervalInSecs = 1800

In limits.conf on your Search Head:

[scheduler]
auto_summary_perc = 100
max_searches_perc = 75
#https://www.rfaircloth.com/2017/12/12/tuning-splunk-when-max-concurrent-searches-are-reached/
[search]
# default base_max_searches value is 6: increase by 10 until utilization on IDX or SH is at 60% CPU/memory starting with 20
base_max_searches = 20

In savedsearches.conf on your Search Head:

[default]
schedule_window = auto
allow_skew = 5m

In server.conf on your Indexers:

#https://static.rainfocus.com/splunk/splunkconf18/sess/15356516230880011oaO/finalPDF/How-To-Solve-Problem-2142_1538789253815001reGC.pdf
[general]
parallelIngestionPipelines = 3

Get as much RAM as you can for your Search Head.
Get a new server and move your DS function onto that so you Search Head does not do DS.

[somesoni2]

This is a small capacity search head, if you've large search load (adhoc searching, dashboarding, alert and reports). If you look at section "Reference host specification for single-instance deployments" and "Dedicated search head" on the Reference Hardware page I shared, you would see what Splunk recommends. You can use your current box as Deployment server (as that role doesn't require high configuration) and create a new dedicated search head with recommended (or better than current) configuration.

You can use following docs to migrate your Search head related contents
(make sure to exclude all deployment server related configurations i.e. serverclass from migration)
https://docs.splunk.com/Documentation/Splunk/7.2.4/Installation/MigrateaSplunkinstance#How_to_migrat...
https://wiki.splunk.com/Deploy:Migrating_a_Splunk_Install

[somesoni2]

Are you using deployment server as search head as well? Could you provide more details on your current environment and it's HW configurations?

I would also recommend, reading this for understanding/planning capacity of your Splunk servers:
https://docs.splunk.com/Documentation/Splunk/7.2.4/Capacity/Referencehardware

[pdantuuri0411]

Yes, The current deployment server is also our search head.

Hardware configuration -

Linux RHEL 2.6.32-754.9.1.el6.x86_64
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 4
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 26
Model name: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
Stepping: 4
CPU MHz: 2399.998
BogoMIPS: 4799.99
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 35840K
NUMA node0 CPU(s): 0-3