Kong ACME Plugin {"message":"failed to update certificate: acme directory request failed: 20: unable to get local issuer certificate"}

Installation/Setup
#1

Having issues with acme plugin to work on docker with cassandra backend with env var
KONG_LUA_SSL_TRUSTED_CERTIFICATE=/etc/ssl/certs/ca-certificates.crt

We have kong set up to listen on port 80 and have confirmed that the response of:

curl KONG_IP/.well-known/acme-challenge/x -H “host:DOMAIN” is Not Found

Getting the error when trying to do the sanity check:

curl http://localhost:8001/acme -d host=subdomain.fake.com -d test_http_challenge_flow=true
{“message”:“failed to update certificate: acme directory request failed: 20: unable to get local issuer certificate”}

Here are logs from the request:

020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event(): worker-events: handling event; source=dao:crud, event=create, pid=nil, data=table: xxxxxxxxxx
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cache.lua:307: invalidate_local(): [DB cache] invalidating (local): 'acme_storage:kong_acme:update_lock:subdomain.fake.com::::'
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_db_cache, pid=7394, data=acme_storage:kong_acme:update_lock:subdomain.fake.com::::
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cache.lua:323: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'acme_storage:kong_acme:update_lock:subdomain.fake.com::::'
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=acme_storage, pid=nil, data=table: 0x7fd88af30dd8
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=acme_storage:create, pid=nil, data=table: 0x7fd88af30dd8
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] pkey.lua:199: load_key(): load key using fmt: *, type: *
2020/07/27 18:49:11 [info] 7394#0: *23699 [lua] pkey.lua:221: load_key(): jwk decode failed: error decoding JSON from JWK: Expected value but found invalid number at character 1, client: 127.0.0.1, server: kong_admin, request: "POST /acme HTTP/1.1", host: "localhost:8001"
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] pkey.lua:255: load_key(): pkey.new:load_key: loaded pkey using function PEM_read_bio_PrivateKey
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event(): worker-events: handling event; source=dao:crud, event=delete, pid=nil, data=table: xxxxxxxxxx
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cache.lua:307: invalidate_local(): [DB cache] invalidating (local): 'acme_storage:kong_acme:update_lock:subdomain.fake.com::::'
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_db_cache, pid=7394, data=acme_storage:kong_acme:update_lock:subdomain.fake.com::::
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cache.lua:323: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'acme_storage:kong_acme:update_lock:subdomain.fake.com::::'
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] cluster.lua:476: next_coordinator(): [lua-cassandra] load balancing policy chose host at 172.0.0.1
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=acme_storage, pid=nil, data=table: xxxxxxxxxx
2020/07/27 18:49:11 [debug] 7394#0: *23699 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=acme_storage:delete, pid=nil, data=table: xxxxxxxxxx
2020/07/27 18:49:11 [info] 7394#0: *23699 client 127.0.0.1 closed keepalive connection
#2

Hi @miguel.d, which docker image version and distro you are using?

#3

I actually built the docker image from a base ubuntu xenial image and compiled kong 2.0.5 there as we had to add some nginx mods. Also added all openresty 1.15.8.2 patches through https://openresty.org/download/

#4

If it’s ubuntu then the ca cert path looks correct. Could you share the full docker env vars, args passed to container and kong.conf if there’s any?

#5

I have everything set up as env vars so no kong.conf

KONG_ADMIN_LISTEN	0.0.0.0:(redacted)
KONG_ADMIN_LISTEN_SSL	0.0.0.0:(redacted)
KONG_CASSANDRA_CONTACT_POINTS	111.111.111(redacted)
KONG_DATABASE	cassandra
KONG_DEBUG_LEVEL	debug
KONG_LOG_LEVEL	debug
KONG_LUA_SSL_TRUSTED_CERTIFICATE	/etc/ssl/certs/ca-certificates.crt
KONG_NGINX_PROXY_LUA_SSL_TRUSTED_CERTIFICATE	/etc/ssl/certs/ca-certificates.crt
KONG_NGINX_PROXY_PROXY_BUFFER_SIZE	16K
KONG_NGINX_PROXY_PROXY_BUFFERS	8 16K
KONG_PLUGINS	bundled,acme
KONG_PROXY_LISTEN	0.0.0.0:80
#6

@miguel.d I vaguely remember there was an update in the ca-certificates package in xenial that affects let’s encrypt CA. Could you check if your base image is update to date?
You can docker exec into the container and see if there’s an error running wget https://acme-v02.api.letsencrypt.org/directory -O -

#7

Ran that command and everything was fine:

wget https://acme-v02.api.letsencrypt.org/directory -O -
--2020-07-29 19:36:45--  https://acme-v02.api.letsencrypt.org/directory
Resolving acme-v02.api.letsencrypt.org (acme-v02.api.letsencrypt.org)... 172.65.32.248, 2606:4700:60:0:f53d:5624:85c7:3a2c
Connecting to acme-v02.api.letsencrypt.org (acme-v02.api.letsencrypt.org)|172.65.32.248|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 658 [application/json]
Saving to: 'STDOUT'

-                                         0%[                                                                              ]       0  --.-KB/s               {
  "M5IPKDa7KFE": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
-                                       100%[=============================================================================>]     658  --.-KB/s    in 0s

2020-07-29 19:36:46 (85.8 MB/s) - written to stdout [658/658]
#8

Hi, I have the same issue, none of the certificates kong managed got renewed ;-(

I’ve upgraded to 2.1.1, tried with both alpine and ubuntu images

curl ${KONG_URL}/acme -d host=myhost.com

give me:

{"message":"failed to update certificate: acme directory request failed: 20: unable to get local issuer certificate"}
#9

@lsbardel there’s currently a bug in acme plugin that in database mode the certificate is not correctly renewed. you can upgrade plugin version to 0.2.9 or wait for next version of kong release. Or you can use dbless mode.

But your error seems irrelevant to the bug and rather a config related issue. Could you share your kong.conf or environment variables?

#10

@miguel.d I would suggest test your setup with official kong images once to exclude the possibility of configuration error. Though from a glance i can’t tell if there’s anything that seems incorrect.

© 2019 Kong Inc.    Terms  •  Privacy  •  FAQ