Acme plugin doesn't work for production URL

dlmn · August 14, 2021, 3:01pm

I’m using a dockerized Kong 2.5.0 instance with the acme plugin. With the staging URL

https://acme-staging-v02.api.letsencrypt.org

The new certificate can be triggered with the command
curl --url https://my-host.com --header 'Host: my-host.com' -k

When I change the acme plugin config to the following URL (let’s encrypt production mode):
https://acme-v02.api.letsencrypt.org/directory

the kong instance logs the following error:

handler.lua:109 failed to update certificate: could not create certificate: error checking challenge: challenge invalid: http-01: invalid: Invalid response from https://my-host.com/.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0 [x.x.x.x]: "<!doctype html><html lang=\"en\"><head><meta charset=\"utf-8\"><link href=\"/favicon.ico\" rel=\"icon\"><meta content=\"width=device-widt", context: ngx.timer, client: 144.2.120.252, server: 0.0.0.0:8443

Here is the log:
18.159.196.172 - - [14/Aug/2021:14:52:49 +0000] "GET /.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0 HTTP/1.1" 302 96 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 18.159.196.172 - - [14/Aug/2021:14:52:49 +0000] "GET /.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0 HTTP/1.1" 200 1002 "http://my-host.com/.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 52.39.4.59 - - [14/Aug/2021:14:52:49 +0000] "GET /.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0 HTTP/1.1" 302 96 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 64.78.149.164 - - [14/Aug/2021:14:52:49 +0000] "GET /.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0 HTTP/1.1" 302 96 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 18.116.86.117 - - [14/Aug/2021:14:52:49 +0000] "GET /.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0 HTTP/1.1" 302 96 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 52.39.4.59 - - [14/Aug/2021:14:52:50 +0000] "GET /.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0 HTTP/1.1" 200 1002 "http://my-host.com/.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 18.116.86.117 - - [14/Aug/2021:14:52:50 +0000] "GET /.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0 HTTP/1.1" 200 1002 "http://my-host.com/.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 64.78.149.164 - - [14/Aug/2021:14:52:50 +0000] "GET /.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0 HTTP/1.1" 200 1002 "http://my-host.com/.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 2021/08/14 14:52:52 [error] 25#0: *11993423 [kong] handler.lua:109 failed to update certificate: could not create certificate: error checking challenge: challenge invalid: http-01: invalid: Invalid response from https://my-host.com/.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0 [x.x.x.x]: "<!doctype html><html lang=\"en\"><head><meta charset=\"utf-8\"><link href=\"/favicon.ico\" rel=\"icon\"><meta content=\"width=device-widt", context: ngx.timer, client: 144.2.120.252, server: 0.0.0.0:8443

Any thoughts appreciated. Thanks!

fomm · August 15, 2021, 7:04am

"<!doctype html><html lang=\"en\"><head><meta charset=\"utf-8\"><link href=\"/favicon.ico\" rel=\"icon\"><meta content=\"width=device-widt",

This seems to suggest https://my-host.com/.well-known/acme-challenge/S6JTWEIrpUIHk2H7VHUjh0UnzuTCH9WvW6D9iHSiSD0 got a webpage returned.

How do you set up your service/route?

dlmn · August 16, 2021, 9:27pm

Thank you for your reply. Indeed it seems that not the right route is chosen. Here is the configuration:

_format_version: "1.1"
services:
- connect_timeout: 60000
  host: 10.10.10.55
  name: acme-dummy
  path: /
  port: 65535
  protocol: http
  read_timeout: 60000
  retries: 5
  write_timeout: 60000
  routes:
  - name: acme-dummy
    paths:
    - /.well-known/acme-challenge
    path_handling: v0
    preserve_host: true
    protocols:
    - http
    - https
    regex_priority: 200
    strip_path: false
    https_redirect_status_code: 426
    request_buffering: true
    response_buffering: true
- connect_timeout: 60000
  host: 10.10.10.55
  name: ui-public
  path: /
  port: 1080
  protocol: http
  read_timeout: 60000
  retries: 5
  write_timeout: 60000
  routes:
  - hosts:
    - my-host.com
    - www.my-host.com
    name: ui-public
    paths:
    - /
    path_handling: v0
    preserve_host: true
    protocols:
    - http
    - https
    regex_priority: 200
    strip_path: true
    https_redirect_status_code: 426
    request_buffering: true
    response_buffering: true
    plugins:
    - name: pre-function
      config:
        access:
        - return function() local scheme = kong.request.get_scheme() if string.match(kong.request.get_host(),"www.")
          or scheme == "http" then local host = kong.request.get_host() local query
          = kong.request.get_path_with_query() local newhost = host.gsub(host,"www.","")
          local url = "https://" .. newhost .. query kong.response.set_header("Location",url)
          return kong.response.exit(302, url) end end
        body_filter: []
        certificate: []
        functions: []
        header_filter: []
        log: []
        rewrite: []
      enabled: true
      protocols:
      - grpc
      - grpcs
      - http
      - https
plugins:
- name: acme
  config:
    account_email: my-email@my-host.com
    api_uri: https://acme-staging-v02.api.letsencrypt.org
    cert_type: rsa
    domains:
    - my-host.com
    - www.my-host.com
    eab_hmac_key: null
    eab_kid: null
    fail_backoff_minutes: 1
    renew_threshold_days: 30
    storage: shm
    storage_config:
      consul:
        host: null
        https: false
        kv_path: null
        port: null
        timeout: null
        token: null
      kong: {}
      redis:
        auth: null
        database: null
        host: null
        port: null
      shm:
        shm_name: kong
      vault:
        host: null
        https: false
        kv_path: null
        port: null
        timeout: null
        tls_server_name: null
        tls_verify: false
        token: null
    tos_accepted: true
  enabled: true
  protocols:
  - grpc
  - grpcs
  - http
  - https
- name: bot-detection
  config:
    allow: []
    deny: []
  enabled: true
  protocols:
  - grpc
  - grpcs
  - http
  - https
- name: cors
  config:
    credentials: false
    exposed_headers: null
    headers: null
    max_age: null
    methods:
    - GET
    - HEAD
    - PUT
    - PATCH
    - POST
    - DELETE
    - OPTIONS
    - TRACE
    - CONNECT
    origins:
    - https://my-host.com
    - https://www.my-host.com
    preflight_continue: false
  enabled: true
  protocols:
  - grpc
  - grpcs
  - http
  - https
- name: response-transformer
  config:
    add:
      headers:
      - 'Cache-Control: no-cache'
      - Strict-Transport-Security:max-age=31536000; includeSubDomains; preload
      - X-Content-Type-Options:nosniff
      - 'X-XSS-Protection: 1; mode=block'
      - 'Content-Security-Policy: default-src ''none''; script-src ''self''  ''sha256-pUCL9ydZHHNKjJSc2boW7NtjfJscMN2q1feiGe2sZZA=''
        *.crisp.chat https://script.hotjar.com https://static.hotjar.com;  style-src
        ''self'' data: ''unsafe-inline'' *.crisp.chat https://*.googleapis.com https://*.gstatic.com;  img-src
        ''self'' data: *.crisp.chat https://script.hotjar.com;  font-src ''self''
        data: *.crisp.chat https://script.hotjar.com https://fonts.googleapis.com
        https://*.gstatic.com;  connect-src ''self''  wss://client.relay.crisp.chat  *.crisp.chat
        https://*.hotjar.com:* https://vc.hotjar.io:* wss://*.hotjar.com blob: data:;  media-src
        ''self'' data:;  frame-src ''self'' *.crisp.chat https://vars.hotjar.com;  frame-ancestors
        ''self'';  form-action ''self'';  block-all-mixed-content;  base-uri ''self'';  manifest-src
        ''self'''
      json: []
      json_types: []
    append:
      headers: []
      json: []
      json_types: []
    remove:
      headers:
      - server
      json: []
    rename:
      headers: []
    replace:
      headers: []
      json: []
      json_types: []
  enabled: true
  protocols:
  - grpc
  - grpcs
  - http
  - https

The weird thing is that on some environments with the same config (different hosts, etc.) it works. And also sometimes it works when I remove the routes and re-adds them with deck sync. And then it even works when the challenge returns the web content or a 502 instead a 404.

Still want to reproduce the error and understand exactly what’s going on.

Thanks for your help

dlmn · August 16, 2021, 9:49pm

I think I found the issue based on your comments. The acme-dummy path has no restriction to the host names. It seems that the priority is changed in that case and the root / path wins because of the presence of host restrictions?

This doesn’t explain the weird behaviour of staging vs prod config but at least now it seems to work.

By the way are the certificates supposed to be updated automatically by the plugin?

Thanks a lot.

fomm · August 17, 2021, 2:02am

Question 1:
It seems that the priority is changed in that case and the root / path wins because of the presence of host restrictions?

This is expected, as mentioned on the official doc

Kong allows for quite some flexibility by allowing two or more Routes to be configured with fields containing the same values - when this occurs, Kong applies a priority rule.

The rule is: when evaluating a request, Kong will first try to match the Routes with the most rules

Because your / route has a host name which means it has more rules.

Question 2:
Are the certificates supposed to be updated automatically by the plugin?

Yes. You can also change how many days before cert expiry you want to renew you certificate via config.renew_threshold_days.

Topic		Replies	Views
ACME/Let's Encrypt on Kong[SOLVED] Questions	0	1099	December 11, 2020
Kong acme plugin not working Installation/Setup	2	840	June 16, 2020
Kong upstream SSL mTLS Questions	16	7945	November 13, 2019
Kong API Gateway SSL/TLS Certificates Questions	1	2406	May 4, 2022
Can I use kong for proxying https request to my upstream if the certificate is locate in the upstream wich is an nginx image that handle certificate with certbot? Questions	1	360	January 30, 2021

Acme plugin doesn't work for production URL

Related Topics