I am trying to get keycloak running behind a kong ingress controller in kubernetes. The issue I am having is kong will not launch on the containers port 443 “containerPort: 443” and when kong passes the request to keycloak, it has a “x-forwarded-port”:“8443” header set. Keycloak tries to redirect everything it sees to the x-forwarded-port, which is not actually the port of the load balancer which is listening to normal 443. I have configured kong to use proxy_protocol, that doesn’t set the x-forwarded-port correctly and I have tired using the request-transformer, but that won’t change the x-forwarded-port header. Any suggestions?
While I’m not familiar with Keycloak, I could find the following documentation:
I’d suggest to look for a setting in Keycloak which can override the
x-forwarded-port value that it sees.
Thanks so much for the reply, I have been trying to fix it on the keycloak side as well. The helm chart for keycloak injects those settings at startup, but they seem to assume the proxy is on 443. One question I have is why is Kong not respecting the proxy_protocol server port when it creates the x-forwarded-port header? real_ip_header does work.
env: trusted_ips: "0.0.0.0/0,::/0" proxy_listen: 0.0.0.0:8080 proxy_protocol, 0.0.0.0:8443 ssl proxy_protocol real_ip_header: proxy_protocol real_ip_recursive: on
I had solved this before by just setting the
containerPort: 443 for Kong, but I assume with the 1.0 release, Kong was moved to the kong user and can’t open those privileged ports anymore.
I think it does respect it but the problem here is the first L7 hop in the request path is Kong, meaning, there are no
X-forwarded-* set and hence Kong assumes that it is the first proxy that is sitting between client and upstream.
proxy_protocol only provides the client IP, which is correctly set as
This is happening because of a mix up of L4 and L7 request metadata. proxy_protocol is at the L4 layer and then
x-forwarded-* are L7 HTTP magic.
The helm chart for keycloak injects those settings at startup, but they seem to assume the proxy is on 443.
Is that not what you really want? Let keycloak assume that the proxy is at 443 and then not respect the
Yes, that would be great, but keycloak assumes the x-forwarded-port is correct and port 443. Seems to tightly coupled to the proxy config to me. But regardless of keycloak, you are absolutely right. I have HAProxy doing tcp forwarding because I want cert-manager in kube to take care of certs, not HAProxy, so there is no x-forwarded-port header, but I could see the PROXY_PROTOCOL string and it looks like this.
PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"
Which includes PROXY_PORT and that is correct at 443. Is there anything in kong that can grab the PROXY_PORT field like it does for
CLIENT_IP -> x-forwarded-for via the
real_ip_header: proxy_protocol setting
Or if there was just some way to hack the
x-forwarded-port: 443 that I haven’t seen, that would solve my problem as well.
apiVersion: configuration.konghq.com/v1 kind: KongPlugin metadata: name: x-forwarded-port namespace: keycloak plugin: request-transformer config: append: headers: - "X-Forwarded-Port: 443"
Doesn’t work for some reason.
Hmmm, It could be the
host header, but still the same issue.
----------------------------REQUEST--------------------------- URI=/auth/admin/ characterEncoding=null contentLength=-1 contentType=null header=X-Real-IP=<REDACTED> header=Accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 header=Accept-Language=en-us header=Accept-Encoding=br, gzip, deflate header=User-Agent=<REDACTED> header=Connection=keep-alive header=X-Forwarded-Proto=https header=X-Forwarded-Port=8443 header=X-Forwarded-For=<REDACTED> header=Referer=https://auth.my-domain.com/auth/ header=Host=auth.my-domain.com:8443 header=X-Forwarded-Host=auth.my-domain.com locale=[en_US] method=GET protocol=HTTP/1.1 queryString= remoteAddr=/<REDACTED> remoteHost=<REDACTED> scheme=https host=auth.my-domain.com:8443 serverPort=8443 isSecure=true --------------------------RESPONSE-------------------------- contentLength=0 contentType=null header=Connection=keep-alive header=Location=https://auth.my-domain.com:8443/auth/admin/master/console/ header=Content-Length=0 header=Date=Fri, 21 Jun 2019 07:05:28 GMT status=302
That’s a bug in Kong Ingress Controller 0.4.0, please use 0.5.0-rc0 if you can to workaround this issue.
I am still having this problem on the very latest versions, now with Jenkins which also cares about the
x-forwarded-port being set correctly. Kong always wants to set this to it’s own port and does not care about what the load balancer in front tells it via proxy protocol, though it does set
APP VERSION: 2
Has anyone found a fix for this?
The only possible fix is to give kong the permission to listen on port 80 and 443. We are tracking a ticket internally to fix this. There is currently no way of overriding the port mapping in Kong.
Thanks so much Harry, you’re awesome!
How to give kong the permission to listen on port 80 and 443?
Thanks and regards,
Kong 2.1.0 is now released.
This problem can be solved using port_map config option:
I redeployed Kong with Kong: v2.1 and Kong-ingress-controller: v0.9.1 on Kubernetes cluster
but still keycloak is not accessible behind Kong v2.1
and giving below error:
“message”:"An invalid response was received from the upstream server "
with error code: 502
What Changes I am missing here?
keycloak deployment yaml:
- name: keycloak
- name: DB_ADDR
- name: DB_DATABASE
- name: DB_PASSWORD
- name: DB_USER
- name: DB_VENDOR
- name: KEYCLOAK_PASSWORD
- name: KEYCLOAK_USER
- name: PROXY_ADDRESS_FORWARDING
- containerPort: 8443
- name: init-postgres-keycloak-service
command: [‘sh’, ‘-c’, ‘until nslookup postgres-keycloak; do echo waiting for postgres-keycloak; sleep 2; done;’]
Please help me with the issue.
Thanks and regards,