How to handle SSL certificate verification for REST API calls from a Splunk app/add-on?

[snargleplax]

I'm developing a Splunk Add-on, and use the REST API in a couple of places to do stuff like look up config values and items from storage/passwords. In testing, I've encountered the problem that I get SSL certificate verification errors because my local test instance of Splunk doesn't have a valid cert set up. I'm wondering about the best way to handle this; not so much just for my own testing, but potentially for users of my add-on who may not have a cert set up on their installation (even if they should, I don't control that and still want my add-on to work).

I've found lots of information about configuring certificates for a given Splunk installation, but nothing about this issue. Looking at some other apps/add-ons that exist, it seems a lot of them simply disable certificate validation for those calls. Is that the best we can do? Ideally, I'd like to require cert validation if the Splunk installation is set up to allow for that, only falling back to disabling verification if that's not possible. Is there some way to get useful information about this for my code to act on, based on the server it's running in? I'm just not crazy about the idea of disabling cert verification across the board simply because some environments may not have the requisite setup. I get that it's probably not the most enormous exposure, since it's Splunk making a REST call to itself, but I still prefer to avoid introducing anything that smells like security exposure.

[frobinson_splun · Accepted solution]

Hi @snargleplax,
I've consulted with some colleagues on this question. In general, validating certificates will require you to know what certificates you are validating against. Once you have these, you can add the validation. There isn't any more specific advice to offer, but hopefully this helps.

[snargleplax]

I think you're answering a different question than I'm asking. I think I may have erred in using the term "validate" rather than "verify," as what I'm referring to is the certificate verification that takes place in automated fashion as part of SSL negotiation. SSL tools and libraries (such as curl, and Python's "requests" library) enable verification by default but provide a flag to disable it if necessary.

My problem is not with how to do that verification (what your answer seems to be about, though I still find it odd since the cert is provided by the server as part of SSL negotiation) -- I know how to do that, as it happens by default with these tools and libraries. Rather, my problem is that I need to recognize situations in which verification should be disabled.

Such situations would occur any time there's a legitimate need to make REST calls to a Splunk installation that doesn't have a valid SSL cert configured. I'm thinking of two categories:

1. Local testing during add-on development.
2. Installation of my add-on by an end user who, for whatever reason, has not chosen to set up their Splunk installation on a host with a valid SSL cert.

I can work around #1 through temporary hackery, but I'm concerned about #2. If I don't provide some way to support such users, they'll be unable to use my add-on. It appears that many existing apps/add-ons address this issue by simply disabling SSL cert verification in all cases, but as I mentioned this seems like a poor security compromise to undertake.

What I'd consider ideal would be if, when invoking the add-on, Splunk provided some indication that the Splunk app server does not have a valid cert available, so verification should not be attempted. I'm guessing no such means exists, and it seems unfortunate because I believe this problem would apply to any app/add-on that needs to make REST calls back to the Splunk server. Surely this is a common use case.

After discussing it within my team, the best workaround we've got is to add a toggle based on an environment variable. That means that, if an end user wants to use our add-on but doesn't have a valid cert set up, they'd have to set up that environment variable when starting up Splunk. It's workable, but kind of an ugly thing to have to ask users to do. I'd love to hear any ideas to improve upon this approach.

[ineeman]

We're using "verification" and "validation" in the same meaning here - so Fiona's
answer was actually in line with what you are asking. Let me try and clarify
some of the difficulties here so that we're on the same page, and hopefully
it will explain why we don't have a good answer to your question.

As you know, Splunk allows administrators to set any certificates they so desire.
The notion of a "valid SSL cert" is in the eyes of the beholder. For example,
there is nothing "invalid" or "unverifiable" with the default certificates
that ship with Splunk - they are simply self-signed, meaning they won't be recognized
by any system that does not have the Splunk CA certificate installed in its own
root trust chain.

Additionally, it's very common for our customers to use their own internal
CAs and thus internal self-signed certificates, which would also fail "validation"/"verification",
as they are not in the trust chain.

When you say you want to verify the certificate, it can be done in one of two
ways:

1. Use the normally trusted certificates (e.g. what httplib2 does by default) to
   verify against, which will fail if you have a self-signed certificate and did not add it to that
   list, even though from a customer's perspective there is nothing wrong with
   that cert.

2. Pass the certificate to validate against, which would mean you need to know
   what that certificate is (but then you can ensure you're talking to the right
   server).

I think (1) is fraught with peril, as you'll continuously run into cases where
a certificate is "just fine", but you'll mark it as invalid. I think (2) is the
correct thing to do, but it means you need to be able to know which certificates
to verify against, which means you need to read that from the Splunk configuration.

This is also fraught with peril, for a couple of reasons: you need to access
this configuration from disk, as I do not believe it's exposed via REST (though
I am not sure, it may be in /server/settings). You'll then need to be able to access
those certificates (or at least the CA), which again may not be possible (and
even if it is possible today may not always be possible)

In short, I think the best way for you to do this would be:

1. Have a parameter in your app (e.g. in your conf files) that enables/disables
   certificate validation.

2. Have a parameter (or set of parameters) that the administrator can specify
   what certificates to validate against, and then you can do that validation properly.

Hopefully this makes sense.

P.S. I used "validate" and "verify" interchangeably here.

[snargleplax]

Thanks, that clear things up greatly. Seems like the approach we're taking (toggle via a documented environment variable) is as good as any.

[frobinson_splun]

Ok, thanks for the additional details @snargleplax. I'll see if there's anything else we can recommend.