Kong upstream SSL mTLS

What would be great is if there was a simple example of getting upstream ssl verification working on Kong v1.3 via docker.
Has anyone out there had success with this? Would be great if there was a best practice or walkthrough out there.

There seems to be a crossover of what should be configured via the admin api and what should be in configuration when it comes to proxy/ssl certs.

Trying kong v1.3 - and getting 502’s when configuring n upstream with ssl_verify on.

My assumption was that I should do the following.

  1. posting the client cert + key to /certificates
  2. associating the right “client_certificate.id” to the service
  3. posting ca for the upstream cert/key combo into /ca_certificates

Muddying the waters are all the kong vars required and I’ve assumed that these are needed, maybe as fallbacks or defaults?
eg
KONG_CLIENT_SSL: “on”
KONG_CLIENT_SSL_CERT: “/mnt/certs/client.crt”
KONG_CLIENT_SSL_CERT_KEY: “/mnt/certs/client.key”
KONG_SSL_CERT: “/mnt/certs/server.crt”
KONG_SSL_CERT_KEY: “/mnt/certs/server.key”

Targets/healthchecks
It’s unclear whether healthchecks.active.https_verify_certificate is using the ca cert provided to /ca_certificates as that fails. So I’ve set that to false and healthchecks work.

Currently cant get past this error.

upstream SSL certificate verify error: (20:unable to get local issuer certificate) while SSL handshaking to upstream,

Have tested manually with curl to the upstream, with all the same certs loaded. Will keep trying, but thought I’d ask around the community :slight_smile:

I’ve got this working, might have just been a permissions issue on the files i was using. Recreating from scratch including the db has worked.
Passing client cert to use for upstream looks like it could be done 2 ways
Either passing the env vars
KONG_CLIENT_SSL: “on”
KONG_CLIENT_SSL_CERT: “/mnt/certs/client.crt”
KONG_CLIENT_SSL_CERT_KEY: “/mnt/certs/client.key”
or not passing those vars and using the new method - posting the client cert + key to to the /certificates url and then patching the service eg
curl -XPOST http://localhost:8002/certificates/ -H ‘Content-Type: application/json’ -d “{“cert”: “(cat client.crt)\", \"key\": \"(cat client.key)”}”
curl -XPATCH http://localhost:8002/services/"{serviceid}" -H 'Content-Type: application/json' -d "{\"client_certificate\": {\"id\":\"{certid)"}}"

2 Likes

Thanks for sharing. Mutual TLS to upstreams is something we intend to test too soon so any guides are helpful!

Client ssl verification is working (upstream nginx likes the client cert offered by kong)
Cant seem to get kong to verify the upstream client cert by posting to /ca_certificates, just ignores it. Gives 200s no matter whats posted here.

Tried injecting nginx_http directives, no luck here either,

injected nginx_http_* directives

proxy_ssl_verify_depth 4;
proxy_ssl_trusted_certificate /mnt/certs/ca.crt;
ssl_protocols TLSv1.1 TLSv1.2 TLSv1.3;
proxy_ssl_verify on;

loading the correct cert to /mnt/certs/ca.crt gives 502s.
upstream SSL certificate verify error: (20:unable to get local issuer certificate) while SSL handshaking to upstream.

Supposedly my colleagues got mutual TLS working today and verified(I haven’t studied what they did).

But my current understanding based on this:
https://konghq.com/blog/kong-1-3-released/

Is you can set the client_cert and private key on a certificates resource:

Then set that certificates resource to the kong service.

But where does the truststore for Kong come into play on these upstream connections? Is the truststore also granular now for mutual TLS to upstreams? I didn’t see much info on that bit.
Combing the Kong template and lua environment variables maybe all upstream Mutual TLS connections rely on:

Alternatively Kong also has a: https://docs.konghq.com/1.3.x/admin-api/#ca-certificate-object

Which seems like a resource that could ideally serve as each services granular truststore if desired. The docs even states “These objects are used by Kong to verify the validity of a client OR server certificate.”. But there is no new service element that seems to tie these two together. So that leaves me with my first guess. Would be good if a Kong engineer could clear up the truststore bit (and we get something in the docs that clarifies that point too).

And maybe in a newer Kong version tie the ca_certificates resource into the mutual TLS to upstreams flow with service resources so each service can trust a separate CA or store a separate public key without having to store everything in that one pem for the current lua ca environment variable(if that is how it indeed works now). Also would promote better security because having 1 truststore filled with tons of CAs or Public Keys opens you up for broader attack if just one of those pub certs or w/e gets private key compromised.

Edit - Actually just did some reading, Kong as the client does not need to trust a client cert presented by the server generally, Kong should trust a CA so tho I think it makes sense if its using the existing PEM LUA env variable and doesn’t need further granularity(as it would be tough to maintain and cause breakage every rotation).

Thanks for that, I have also tried with the LUA_SSL_TRUSTED_CERTIFICATE var (by setting the KONG_LUA_SSL_TRUSTED_CERTIFICATE env var).
I set this to a CA cert that doesn’t verify, but requests through Kong keep on working regardless, and this indicated to me that it’s just ignored. Wondering if your colleagues tested with an incorrect ca and proved it fails on this version?

/ # openssl verify -CAfile /mnt/certs/ca.crt /tmp/upstream.crt
error 20 at 0 depth lookup: unable to get local issuer certificate
error /tmp/upstream.crt: verification failed

cheers

What you’re describing seems to be an issue with Kong (client) trusting some Upstream (server) - that’s actually an issue with 1-way SSL, not 2-way.

Client trusting Server is 1-way
Client trusting Server AND Server trusting Client is 2-way

I went and did a quick test. My own Kong instance trusts only the Comodo CA chain, and I was able to make a proxy which successfully routes over HTTPS to a site with a DigiCert CA chain… doesn’t seem like that should work.

TL;DR Agree this is an issue, but with 1-way SSL, not with 2-way SSL.

Basically, if Kong was curl, it would be using curl -k every time.

Yea agree with the TL;DR. Thanks for testing !
May I ask how you loaded the CA Certificate in your setup? nginx directives, lua_ssl_trusted_certificate ,rest to /ca_certificates?

cheers

@Ross_Sbriscia and I are colleagues @england_2022 ,

We have these env vars:

KONG_SSL_CERT:  ..../cert.crt
KONG_SSL_CERT_KEY: ...../privatekey.key
KONG_LUA_SSL_TRUSTED_CERTIFICATE:  ...../ca.pem

Which we know KONG_LUA_SSL_TRUSTED_CERTIFICATE is the path to the certificate authority file for Lua cosockets in PEM format. This certificate will be the one used for verifying Kong’s database connections. <- From the docs , I wasn’t sure if Kongs underlying new libs for mutual TLS were also going to leverage this pem for its reverse proxying

And the other two we know are for TLS Connections for Kong as the server.

Ohhh… Idk why I didn’t just go to the NGINX documentation first… Looks like for Kong to start validating upstream services we just need them to add these directives in the nginx_kong.lua template file:

http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_trusted_certificate

proxy_ssl_trusted_certificate ...../trusted-cert.pem;
proxy_ssl_verify on;
proxy_ssl_verify_depth X; 0 for self-signed or however many intermediaries + root CA's you have in the chain, so like 2 intermediaries + root CA depth would be 3 I think

Inside that PEM should be your CA’s (Root/Intermediaries) for validation of your upstreams. I am actually surprised this has not been a thing in Kong yet. Should be easy to add new ENV Variables to the system and drop in the goodies here below proxy_pass:

Most Kong users can probably use the same PEM they use for KONG_LUA_SSL_TRUSTED_CERTIFICATE env variable they use for DB connections likely if your CA’s with your upstreams == the CA of your db nodes. I suppose some granularity would be nice though to decide which service resources to subject to certificate verification but just having the all or none option would be a good start.

Edit - Oh but reading above you mentioned you did try that… in that case maybe Kongs own TLS logic is causing issue with the nginx directives? Idk… @Ross_Sbriscia lets try these NGX directives at some point in dev on our 1.3 and see if they also don’t work for us.

Just to add that I can also verify that this is also used by the healthcheck library (GitHub - Kong/lua-resty-healthcheck: Healthcheck library for OpenResty to validate upstream service status).

You will need to set LUA_SSL_TRUSTED_CERTIFICATE if you want to set SSL verification on for the healthchecks.

Yep for sure, because that env variable is this under the hood:

Which is used for all the TLS lua-nginx-module TCP connections:

So yeah just tested and had similar findings, by adding these directives its not possible to proxy to anything on Kong 1.3, even if its signed by the CA’s in our PEM or not:

        proxy_pass         $upstream_scheme://kong_upstream$upstream_uri;
        proxy_ssl_trusted_certificate '${{LUA_SSL_TRUSTED_CERTIFICATE}}';
        proxy_ssl_verify on;
        proxy_ssl_verify_depth 3;

Throws something like:

2019/10/11 22:48:56 [error] 32#0: *8525 upstream SSL certificate verify error: (20:unable to get local issuer certificate) while SSL handshaking to upstream, client: 10.xx.xx.xxx, server: kong, request: "GET /api/digicert HTTP/1.1", upstream: "https://xx.xx.xxx.xxx:443/ ", host: "gateway.company.com"

As an HTTP 502

Kong Error

An invalid response was received from the upstream server.

1 Like

Thanks for testing Jeremy
Looks like the issue is related to kong service mesh getting in the way - check out https://github.com/Kong/kong/issues/5112

cheers

1 Like

Yep I see them putting in the good work to fix up the issues there around mesh getting in the way. Good stuff!

Hello everyone,

Thanks for the detailed report! This bug was caused by the Service Mesh feature of Kong which has been turned off by default since 1.4.0rc2. Please give it a try and let us know if it solved your problems:

Best,
Datong

1 Like

Will give it a spin in the next week or two! Sounds like the direction is Service Mesh will get focused into Kuma and Kong will discontinue mesh framework being built out based on what I read:

Which is fine by me :+1:, for now I am still just focused on Gateway functionality anyways.

Reporting here that indeed 1.4 behaves as expected!

Throws a:

2019/11/13 03:12:42 [error] 32#0: *493462 upstream SSL certificate verify error: 
(20:unable to get local issuer certificate) while SSL handshaking to upstream, client: 10.xx.xx.xxx, 
server: kong, request: "PURGE /api/dev/cdm/MyCoolTestResource/v14 HTTP/1.1", upstream: 
"https://172.xxx.x.xx:443/api/new/path ", 
host: "gateway-dev-dc.company.com"

If the api presents a certificate not signed by the truststore CAs. And worked like charm for internal endpoints signed by our CA. Good stuff :slight_smile: .

Config in my template was as follows:

        # Added upstream certificate verification
        proxy_ssl_trusted_certificate '${{LUA_SSL_TRUSTED_CERTIFICATE}}';
        proxy_ssl_verify on;
        proxy_ssl_verify_depth 3;
1 Like