Kong 3.9.0 with ACME plugin throws invalid response / no corresponding response found for

Questions
,
1

Hello,

running Kong 3.9.0 OSS on a host machine, I’m trying to use the ACME plugin respectively the kong-certbot-agent to create/trigger certificate generation.

{
  "next": null,
  "data": [
    {
      "id": "6c2bc4dd-ae31-478f-a0ae-a98cf6c803d1",
      "request_buffering": true,
      "response_buffering": true,
      "https_redirect_status_code": 426,
      "snis": null,
      "protocols": [
        "https"
      ],
      "tags": [],
      "destinations": null,
      "headers": null,
      "methods": null,
      "strip_path": false,
      "hosts": [
        "apinew.itembase.com"
      ],
      "preserve_host": true,
      "service": {
        "id": "b9936b3a-4da5-43f5-b258-69269c238707"
      },
      "path_handling": "v0",
      "name": "connectivity-endpoint",
      "paths": [
        "/connectivity"
      ],
      "created_at": 1742829652,
      "updated_at": 1744043757,
      "regex_priority": 0,
      "sources": null
    },
    {
      "id": "963501ae-f0b8-4cbc-8772-a3b04c51ebff",
      "request_buffering": true,
      "response_buffering": true,
      "https_redirect_status_code": 426,
      "snis": null,
      "protocols": [
        "http"
      ],
      "tags": [],
      "destinations": null,
      "headers": null,
      "methods": null,
      "strip_path": false,
      "hosts": [
        "apinew.itembase.com"
      ],
      "preserve_host": false,
      "service": null,
      "path_handling": "v0",
      "name": null,
      "paths": [
        "/.well-known/acme-challenge"
      ],
      "created_at": 1744042703,
      "updated_at": 1744107533,
      "regex_priority": 0,
      "sources": null
    }
  ]
}

The DNS resolves to the proper publi IP address which is on a LB (keepalived) . The LB forwards the requetss to Kongs private IP (10.0.106.114) on port 8000 (same configuration is set for 8443)

Running a test from a an external host like this

curl -v -H “Host: apinew.itembase.com” http://apinew.itembase.com/.well-known/acme-challenge/test

Results into:

*   Trying 83.141.6.84...
* TCP_NODELAY set
* Connected to apinew.itembase.com (83.141.6.84) port 80 (#0)
> GET /.well-known/acme-challenge/test HTTP/1.1
> Host: apinew.itembase.com
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Tue, 08 Apr 2025 10:38:53 GMT
< Content-Type: application/json; charset=utf-8
< Connection: keep-alive
< Content-Length: 78
< X-Kong-Response-Latency: 8
< Server: kong/3.9.0
< X-Kong-Request-Id: 9030d34f93950a2247bebb1e7cf8027c
<
{
  "message":"Not found",
  "request_id":"9030d34f93950a2247bebb1e7cf8027c"
* Connection #0 to host apinew.itembase.com left intact

I’ve also used another acme client to trigger the creation of the certificate like this

docker run  --rm  -p 80:80 phpdockerio/kong-certbot-agent:3.2.2 ./certbot-agent certs:update   http://10.0.106.114:8001 xx.k@xx.com apinew.itembase.com

and received this

Updating certificates config for apinew.itembase.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for apinew.itembase.com
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. apinew.itembase.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: 83.141.6.84: Invalid response from http://apinew.itembase.com/.well-known/acme-challenge/5MgA1RhIBzfPJldPCP6KqIbfoH2t7cErZTpmJNRM1yo: 404
Certbot error: command status 1, output `["IMPORTANT NOTES:"," - The following errors were reported by the server:","","   Domain: apinew.itembase.com","   Type:   unauthorized","   Detail: 83.141.6.84: Invalid response from","   http:\/\/apinew.itembase.com\/.well-known\/acme-challenge\/5MgA1RhIBzfPJldPCP6KqIbfoH2t7cErZTpmJNRM1yo:","   404","","   To fix these errors, please make sure that your domain name was","   entered correctly and the DNS A\/AAAA record(s) for that domain","   contain(s) the right IP address."," - Your account credentials have been saved in your Certbot","   configuration directory at \/etc\/letsencrypt. You should make a","   secure backup of this folder now. This configuration directory will","   also contain certificates and private keys obtained by Certbot so","   making regular backups of this folder is ideal."]`, domains apinew.itembase.com

Kong error logs are showing this

2025/04/08 10:16:52 [debug] 133175#0: *8233 [lua] compat.lua:29: log_debug(): jwk decode failed: error decoding JSON from JWK: Expected value but found invalid number at character 1, continuing
2025/04/08 10:16:52 [debug] 133175#0: *8233 [lua] compat.lua:29: log_debug(): pkey:load_pem_der: loaded pkey using function PEM_read_bio_PrivateKey
2025/04/08 10:16:52 [debug] 133175#0: *8233 [acme] http-01.lua:45: token is lLMWe12KJSCegK80IDUhZQuk1-MhTEbfv0hzsVvJOkw
2025/04/08 10:16:52 [warn] 133175#0: *8233 [acme] http-01.lua:55: no corresponding response found for lLMWe12KJSCegK80IDUhZQuk1-MhTEbfv0hzsVvJOkw, client: 23.178.112.108, server: kong, request: "GET /.well-known/acme-challenge/lLMWe12KJSCegK80IDUhZQuk1-MhTEbfv0hzsVvJOkw HTTP/1.1", host: "apinew.itembase.com", request_id: "d663a13cb3924e7585f4134267337b10"
2025/04/08 10:16:52 [debug] 133175#0: *8233 [lua] init.lua:24: poll(): worker-events: emulate poll method

I’m using kong as a storage.

I’m not sure why this is giving me the error I see.

I appreciate any support

Best regards
Ramo