Kong ACME Plugin {"message":"failed to update certificate: acme directory request failed: 20: unable to get local issuer certificate"}

Installation/Setup
#1

Having issues with acme plugin to work on docker with cassandra backend with env var
KONG_LUA_SSL_TRUSTED_CERTIFICATE=/etc/ssl/certs/ca-certificates.crt

We have kong set up to listen on port 80 and have confirmed that the response of:

curl KONG_IP/.well-known/acme-challenge/x -H “host:DOMAIN” is Not Found

Getting the error when trying to do the sanity check:

curl http://localhost:8001/acme -d host=subdomain.fake.com -d test_http_challenge_flow=true
{“message”:“failed to update certificate: acme directory request failed: 20: unable to get local issuer certificate”}

Here are logs from the request:

020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event(): worker-events: handling event; source=dao:crud, event=create, pid=nil, data=table: xxxxxxxxxx
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cache.lua:307: invalidate_local(): [DB cache] invalidating (local): 'acme_storage:kong_acme:update_lock:subdomain.fake.com::::'
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_db_cache, pid=7394, data=acme_storage:kong_acme:update_lock:subdomain.fake.com::::
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cache.lua:323: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'acme_storage:kong_acme:update_lock:subdomain.fake.com::::'
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=acme_storage, pid=nil, data=table: 0x7fd88af30dd8
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=acme_storage:create, pid=nil, data=table: 0x7fd88af30dd8
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] pkey.lua:199: load_key(): load key using fmt: *, type: *
2020/07/27 18:49:11 [info] 7394#0: *23699 [lua] pkey.lua:221: load_key(): jwk decode failed: error decoding JSON from JWK: Expected value but found invalid number at character 1, client: 127.0.0.1, server: kong_admin, request: "POST /acme HTTP/1.1", host: "localhost:8001"
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] pkey.lua:255: load_key(): pkey.new:load_key: loaded pkey using function PEM_read_bio_PrivateKey
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event(): worker-events: handling event; source=dao:crud, event=delete, pid=nil, data=table: xxxxxxxxxx
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cache.lua:307: invalidate_local(): [DB cache] invalidating (local): 'acme_storage:kong_acme:update_lock:subdomain.fake.com::::'
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_db_cache, pid=7394, data=acme_storage:kong_acme:update_lock:subdomain.fake.com::::
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cache.lua:323: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'acme_storage:kong_acme:update_lock:subdomain.fake.com::::'
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=acme_storage, pid=nil, data=table: xxxxxxxxxx
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=acme_storage:delete, pid=nil, data=table: xxxxxxxxxx
2020/07/27 18:49:11 [info] 7394#0: *23699 client 127.0.0.1 closed keepalive connection
#2

Hi @miguel.d, which docker image version and distro you are using?

#3

I actually built the docker image from a base ubuntu xenial image and compiled kong 2.0.5 there as we had to add some nginx mods. Also added all openresty 1.15.8.2 patches through https://openresty.org/download/

#4

If it’s ubuntu then the ca cert path looks correct. Could you share the full docker env vars, args passed to container and kong.conf if there’s any?

#5

I have everything set up as env vars so no kong.conf

KONG_ADMIN_LISTEN	0.0.0.0:(redacted)
KONG_ADMIN_LISTEN_SSL	0.0.0.0:(redacted)
KONG_CASSANDRA_CONTACT_POINTS	111.111.111(redacted)
KONG_DATABASE	cassandra
KONG_DEBUG_LEVEL	debug
KONG_LOG_LEVEL	debug
KONG_LUA_SSL_TRUSTED_CERTIFICATE	/etc/ssl/certs/ca-certificates.crt
KONG_NGINX_PROXY_LUA_SSL_TRUSTED_CERTIFICATE	/etc/ssl/certs/ca-certificates.crt
KONG_NGINX_PROXY_PROXY_BUFFER_SIZE	16K
KONG_NGINX_PROXY_PROXY_BUFFERS	8 16K
KONG_PLUGINS	bundled,acme
KONG_PROXY_LISTEN	0.0.0.0:80
#6

@miguel.d I vaguely remember there was an update in the ca-certificates package in xenial that affects let’s encrypt CA. Could you check if your base image is update to date?
You can docker exec into the container and see if there’s an error running wget https://acme-v02.api.letsencrypt.org/directory -O -

#7

Ran that command and everything was fine:

wget https://acme-v02.api.letsencrypt.org/directory -O -
--2020-07-29 19:36:45--  https://acme-v02.api.letsencrypt.org/directory
Resolving acme-v02.api.letsencrypt.org (acme-v02.api.letsencrypt.org)... 172.65.32.248, 2606:4700:60:0:f53d:5624:85c7:3a2c
Connecting to acme-v02.api.letsencrypt.org (acme-v02.api.letsencrypt.org)|172.65.32.248|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 658 [application/json]
Saving to: 'STDOUT'

-                                         0%[                                                                              ]       0  --.-KB/s               {
  "M5IPKDa7KFE": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
-                                       100%[=============================================================================>]     658  --.-KB/s    in 0s

2020-07-29 19:36:46 (85.8 MB/s) - written to stdout [658/658]
#8

Hi, I have the same issue, none of the certificates kong managed got renewed ;-(

I’ve upgraded to 2.1.1, tried with both alpine and ubuntu images

curl ${KONG_URL}/acme -d host=myhost.com

give me:

{"message":"failed to update certificate: acme directory request failed: 20: unable to get local issuer certificate"}
#9

@lsbardel there’s currently a bug in acme plugin that in database mode the certificate is not correctly renewed. you can upgrade plugin version to 0.2.9 or wait for next version of kong release. Or you can use dbless mode.

But your error seems irrelevant to the bug and rather a config related issue. Could you share your kong.conf or environment variables?

#10

@miguel.d I would suggest test your setup with official kong images once to exclude the possibility of configuration error. Though from a glance i can’t tell if there’s anything that seems incorrect.

#11

This is the seup on my dev server

kong.conf

bash-5.0$ cd /etc/kong/
bash-5.0$ tail kong.conf
headers = latency_tokens
plugins = bundled,metablock
nginx_proxy_lua_ssl_trusted_certificate = /etc/ssl/cert.pem
bash-5.0$

env variables

bash-5.0$ env | grep KONG_
KONG_PG_DATABASE=kong
KONG_ADMIN_LISTEN=0.0.0.0:8001
KONG_VERSION=2.1.1
KONG_PG_PASSWORD=kong
KONG_PG_USER=kong
KONG_PG_HOST=database
KONG_DATABASE=postgres
KONG_LOG_LEVEL=debug

If I don’t set the nginx_proxy_lua_ssl_trusted_certificate = /etc/ssl/cert.pem the acme plugin does not work at all. Currently it get certificates but does not renew them which makes it unusable unfortunately.

#12

@lsbardel try use lua_ssl_trusted_certificate instead of nginx_proxy_lua_ssl_trusted_certificate one, as the certificate is created from Admin API not from proxy side. We used to have nginx_proxy_lua_ssl_trusted_certificate in document but it’s now updated to use lua_ssl_trusted_certificate to cover both admin and proxy side.

#13

@Wangchong_Zhou that fixed the issue, many thanks.
I’m able to force renewal.

I’ve switched to env variables in the meantime, so I have

KONG_LUA_SSL_TRUSTED_CERTIFICATE:  /etc/ssl/certs/ca-certificates.crt

using the kong:2.1.1-ubuntu image

I guess for the fix of auto-renewal in db mode will be in the next kong release right?

1 Like
#14

@lsbardel cool. correct it will be in next version, on your can build a custom image on top of the official kong image that adds the 0.2.9 version plugin (https://github.com/kong/kong-plugin-acme).

#15

Good information thanks for sharing
vmware

© 2019 Kong Inc.    Terms  •  Privacy  •  FAQ