SSL certificate for F5 VIP to search head cluster?

[rpquinlan]

We're finishing up our migration from a single search head to a search head cluster. Our company uses F5 load balancers. Per this http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/UseSHCwithloadbalancers , I had the web guys set me up with a VIP that points to our 2 search heads, using layer-7 processing and persistence.

In order to keep the clients from getting the SSL certificate warning every time they log in, I wanted to have a certificate made for the friendly 'splunk.company.com' URL. The web guys are telling me that because Splunk specifies a layer 7 profile, that they can have a cert on the VIP, that it would have to be an individual cert on each search head, which I don't think would prevent the warnings in the browser...

Has anyone else run into this?

[jtacy · Accepted solution]

The load balancer is a fine place to keep a trusted CA-signed certificate and you're referring to an SSL bridging configuration where different certs can be used with the client-side (client to F5) and server-side (F5 to Splunk) connections. I would personally consider 3 VIPs:

VIP 1 - http->https redirect
Port: 80
Action: Redirect to https (requires http profile)

VIP 2 - Splunk UI
Port: 443
SSL Config: Client SSL profile with splunk.company.tld certificate, default server SSL profile (SSL bridging)
Real Servers: splunk1:8443, splunk2:8443, splunk3:8443
Persistence: Insert cookie (requires http profile)
Other Config: OneConnect profile (I use a /0 mask but /32 is fine) (requires http profile)
Health Monitor Type: https
Health Monitor Send String: GET /en-US/account/login HTTP/1.1\r\nHost: splunk.company.tld\r\nConnection: close\r\n\r\n
Health Monitor Receive String: 200 OK

VIP 3 - Splunk API
Port: 8089
SSL Config: Client SSL profile with splunk.company.tld certificate, default server SSL profile (SSL bridging)
Real Servers: splunk1:8089, splunk2:8089, splunk3:8089
Persistence: Source IP with 20 minute timeout
Other Config: OneConnect profile (I use a /0 mask but /32 is fine) (requires http profile)
Health Monitor Type: https
Health Monitor Send String: GET / HTTP/1.1\r\nHost: splunk.company.tld\r\nConnection: close\r\n\r\n
Health Monitor Receive String: 200 OK

This configuration is tested on Splunk 6.5.3. Note that we're using port 8443 instead of the default port 8000 on the Splunk UI. I guess you could omit the 8089 VIP if you don't have clients calling the API. Other than that, this should work out of the box. You can keep your self-signed certificates in Splunk and let your F5 group handle the trusted CA-signed cert. Your users will always see a good cert.

I should add that I'm slightly concerned about future version upgrades affecting the health monitors but that will be caught in staging if it's going to happen. I don't know what, if anything, Splunk recommends for health monitors on a SHC. You could use a TCP or TCP half-open monitor on the F5 if you just want to see if the port is open.

You may also want to deploy an alert_actions.conf like this to your SHC so alerts will be generated with the correct URLs:

[default]
hostname = https://splunk.company.tld

Have fun!

[maniu1609]

@jtacy Your answer to this question is really helpful for many people. Appreciate your effort. I have one doubt here.

I have enabled ssl in the splunk server that means splunk self signed ssl certificate is used.

Now I tried this url https://localhost:8443/ from postman tool with ssl certification validation enabled. But i didn't get 200 OK status instead i got below error:

"Self-signed SSL certificates are being blocked:
Fix this by turning off 'SSL certificate verification' in Settings > General"

Then i disabled ssl certification validation option in the tool and then hit this url again https://localhost:8443/. Now i'm getting 200 OK status.

So my question here is, Will the below health monitor be able to send HTTPS request to server from F5 and return 200 OK response even though we use self signed certification at server side?

Health Monitor Send String: GET /en-US/account/login HTTP/1.1\r\nHost: splunk.company.tld\r\nConnection: close\r\n\r\n

[rpquinlan]

Thanks very much for the detailed response - I'll send this over to our F5 guys!

[rpquinlan]

As an update - everything is working perfectly with the above config.