X-forwarded-port

Questions
1

I am trying to get keycloak running behind a kong ingress controller in kubernetes. The issue I am having is kong will not launch on the containers port 443 “containerPort: 443” and when kong passes the request to keycloak, it has a “x-forwarded-port”:“8443” header set. Keycloak tries to redirect everything it sees to the x-forwarded-port, which is not actually the port of the load balancer which is listening to normal 443. I have configured kong to use proxy_protocol, that doesn’t set the x-forwarded-port correctly and I have tired using the request-transformer, but that won’t change the x-forwarded-port header. Any suggestions?

2

While I’m not familiar with Keycloak, I could find the following documentation:

https://www.keycloak.org/docs/latest/server_installation/index.html#enable-https-ssl-with-a-reverse-proxy

I’d suggest to look for a setting in Keycloak which can override the x-forwarded-port value that it sees.

3

Thanks so much for the reply, I have been trying to fix it on the keycloak side as well. The helm chart for keycloak injects those settings at startup, but they seem to assume the proxy is on 443. One question I have is why is Kong not respecting the proxy_protocol server port when it creates the x-forwarded-port header? real_ip_header does work.

env:
  trusted_ips: "0.0.0.0/0,::/0"
  proxy_listen: 0.0.0.0:8080 proxy_protocol, 0.0.0.0:8443 ssl proxy_protocol
  real_ip_header: proxy_protocol
  real_ip_recursive: on

I had solved this before by just setting the containerPort: 443 for Kong, but I assume with the 1.0 release, Kong was moved to the kong user and can’t open those privileged ports anymore.

4

I think it does respect it but the problem here is the first L7 hop in the request path is Kong, meaning, there are no X-forwarded-* set and hence Kong assumes that it is the first proxy that is sitting between client and upstream.

AFAIK proxy_protocol only provides the client IP, which is correctly set as x-real-ip.

This is happening because of a mix up of L4 and L7 request metadata. proxy_protocol is at the L4 layer and then x-forwarded-* are L7 HTTP magic.

The helm chart for keycloak injects those settings at startup, but they seem to assume the proxy is on 443.

Is that not what you really want? Let keycloak assume that the proxy is at 443 and then not respect the x-forwarded-port header?

5

Yes, that would be great, but keycloak assumes the x-forwarded-port is correct and port 443. Seems to tightly coupled to the proxy config to me. But regardless of keycloak, you are absolutely right. I have HAProxy doing tcp forwarding because I want cert-manager in kube to take care of certs, not HAProxy, so there is no x-forwarded-port header, but I could see the PROXY_PROTOCOL string and it looks like this.

PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"

Which includes PROXY_PORT and that is correct at 443. Is there anything in kong that can grab the PROXY_PORT field like it does for CLIENT_IP -> x-forwarded-for via the real_ip_header: proxy_protocol setting

Or if there was just some way to hack the x-forwarded-port: 443 that I haven’t seen, that would solve my problem as well.

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: x-forwarded-port
  namespace: keycloak
plugin: request-transformer
config:
  append:
    headers:
      - "X-Forwarded-Port: 443"

Doesn’t work for some reason.

6

Hmmm, It could be the host header, but still the same issue.

----------------------------REQUEST---------------------------
               URI=/auth/admin/
 characterEncoding=null
     contentLength=-1
       contentType=null
            header=X-Real-IP=<REDACTED>
            header=Accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
            header=Accept-Language=en-us
            header=Accept-Encoding=br, gzip, deflate
            header=User-Agent=<REDACTED>
            header=Connection=keep-alive
            header=X-Forwarded-Proto=https
            header=X-Forwarded-Port=8443
            header=X-Forwarded-For=<REDACTED>
            header=Referer=https://auth.my-domain.com/auth/
            header=Host=auth.my-domain.com:8443
            header=X-Forwarded-Host=auth.my-domain.com
            locale=[en_US]
            method=GET
          protocol=HTTP/1.1
       queryString=
        remoteAddr=/<REDACTED>
        remoteHost=<REDACTED>
            scheme=https
              host=auth.my-domain.com:8443
        serverPort=8443
          isSecure=true
--------------------------RESPONSE--------------------------
     contentLength=0
       contentType=null
            header=Connection=keep-alive
            header=Location=https://auth.my-domain.com:8443/auth/admin/master/console/
            header=Content-Length=0
            header=Date=Fri, 21 Jun 2019 07:05:28 GMT
            status=302
7

That’s a bug in Kong Ingress Controller 0.4.0, please use 0.5.0-rc0 if you can to workaround this issue.

8

Good afternoon,

I am still having this problem on the very latest versions, now with Jenkins which also cares about the x-forwarded-port being set correctly. Kong always wants to set this to it’s own port and does not care about what the load balancer in front tells it via proxy protocol, though it does set x-forwarded-for correctly.

CHART: kong-1.5.0
APP VERSION: 2

Has anyone found a fix for this?

9

The only possible fix is to give kong the permission to listen on port 80 and 443. We are tracking a ticket internally to fix this. There is currently no way of overriding the port mapping in Kong.

10

Thanks so much Harry, you’re awesome!

11

Hi Harry,

How to give kong the permission to listen on port 80 and 443?

Thanks and regards,
Rahul Salunke

12

Kong 2.1.0 is now released.
This problem can be solved using port_map config option:

13

Hi Harry,

I redeployed Kong with Kong: v2.1 and Kong-ingress-controller: v0.9.1 on Kubernetes cluster

but still keycloak is not accessible behind Kong v2.1
and giving below error:
{
“message”:"An invalid response was received from the upstream server "
}
with error code: 502

What Changes I am missing here?

keycloak deployment yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
creationTimestamp: null
labels:
io.kompose.service: keycloak
name: keycloak
spec:
replicas: 1
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
io.kompose.service: keycloak
spec:
containers:
- name: keycloak
env:
- name: DB_ADDR
value: postgres-keycloak
- name: DB_DATABASE
value: “keycloak”
- name: DB_PASSWORD
value: password
- name: DB_USER
value: keycloak
- name: DB_VENDOR
value: POSTGRES
- name: KEYCLOAK_PASSWORD
value: Pa55w0rd
- name: KEYCLOAK_USER
value: admin
- name: PROXY_ADDRESS_FORWARDING
value: “false”
image: XXXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/breeze/keycloak:latest
ports:
- containerPort: 8443
resources: {}
initContainers:
- name: init-postgres-keycloak-service
image: busybox
command: [‘sh’, ‘-c’, ‘until nslookup postgres-keycloak; do echo waiting for postgres-keycloak; sleep 2; done;’]
restartPolicy: Always

Please help me with the issue.

Thanks and regards,
Rahul Salunke

14

Hi all,
I solved the issue with Keycloak with a serverless function. It works in my case.

config:
  functions:
    - return function()
        if ngx.var.upstream_x_forwarded_port == "8000" then
          ngx.var.upstream_x_forwarded_port = 80
        elseif ngx.var.upstream_x_forwarded_port == "8443" then
          ngx.var.upstream_x_forwarded_port = 443
         end
      end
plugin: post-function

May be it helps.

Best,
Daniel