What are some of the best practices of setting up new Splunk servers?

[cecampbell]

Hello,

We recently created 5 new Splunk servers with Windows Server 2016 installed, our current deployment is, 2 indexers, 2 search heads, with a deployment server, is this still the ideal setup? I am new to Splunk so just want to make sure we are doing best practice.

Our current setup we have Enterprise Security and Core Splunk both on the search heads.

They all 24 GB of RAM and 6cpu and 6 sockets.

Eventually, I would like to migrate the old data to the new servers and would like to know is that something that should be done?

[beatus]

Cecampbell,
I'd highly recommend you engage Professional services for this. It sounds like you're new to Splunk and ES is a very complicated product. Based on the information you've provided so far, I'm very concerned with your deployment and wouldn't recommend going forward with the path you've laid out. Some additional information would be required to make a final judgement, that said my initial reaction is you're on a path for major pain. Some issues I see so far:

• Below minimum specs for CPU (6 socket systems are not a thing, i'm assuming either single socket or dual socket) / Memory
• It sounds like ES is installed on both search heads? That's a big issue if so.
• Windows (Not a deal breaker, but also going to draw flak from others)

Some additional info that would help:
- License size
- Current amount of stored data
- Storage subsystem

Again, I'd HIGHLY recommend engaging Splunk Professional services for this. ES is a complex product, under-sizing it from the get go will be a massive problem. Migrating data is also a complex undertaking with many variables that PS can help with.

[cecampbell]

Hello Beatus,

Thanks for the feedback, I will push our team to give more resources.

The ES is only on the 1 search head.

We initially used PS, and this is the architecture they recommended, but now we are rebuilding the servers.

http://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/SHCwithindexers

[nmiller_splunk]

I'll point out again that you cannot have a 2 member SHC. It's not supported. Secondly, you cannot run ES on a single member of a SHC. All apps must be homogenous across a SHC.

If the intent is to have two separate search heads, one for ES and one for non-ES, then that is workable, depending on ingest and users' adhoc search load, in a 2 SH/2 IDX environment. ES consumes large amounts of search head and indexer resources regardless of the ingest level due to DMAs. You will not be able to get by on minimum system resources and have a positive experience.

[nmiller_splunk]

First off, the minimum requirements for an Enterprise Security search head are 16 physical cores and 32gb of RAM. You should probably start with the following documentation: http://docs.splunk.com/Documentation/ES/5.0.0/Install/DeploymentPlanning and http://docs.splunk.com/Documentation/Splunk/7.0.3/Capacity/ComponentsofaSplunkEnterprisedeployment

[cecampbell]

Thanks for the feedback nmiller, I am unaware they are under sized, our systems team, knew the requirements, but felt as if it was too much resources and advised they will add additional resources once they see that it is needed :(.

I am following the below document, and have 2 search heads, and 2 indexers, and a deployer.

http://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/SHCwithindexers

[nmiller_splunk]

Your systems do not meet the minimum specifications for core Splunk, either. You need to have a serious chat with your systems team, as this will be a very poor experience. Splunk on virtual environments must have reserved resources, and with the negative performance impact of the Meltdown/Spectre patches, having more than minimum resources to run Splunk is generally necessary unless you have a very lightly used environment.

Next, you cannot have a SHC with only two members. This is 100% not supported.

Third, if you are not familiar with Enterprise Security or Search Head Clustering, you will have an extremely steep learning curve implementing both.

I highly recommend that you step back, read all documentation regarding Enterprise Security and capacity planning, and then reassess your architecture and expertise level before continuing with your current plans.

The majority of our customers do not implement Enterprise Security without a professional services engagement.