HTTP to HTTPS redirect with kong-ingress and TLS-offloading

Kubernetes
#1

Hi all.

I’m having issues setting up HTTP to HTTPS redirect on the kong ingress while using AWS ELB (classic) and HTTPS listener for TLS offloading. With the current configuration no matter whether the request arrives via HTTP or HTTPS kong issues 302 redirect. By the looks of it Kong is not handling the X-Forward-Proto header supplied by the ELB.

I have followed the posts in the forum about similar issue (Redirecting HTTP to HTTPS) but so far it doesn’t seem that anyone had success implementing proper HTTP to HTTPS redirect with TLS offloading or am I configuring Kong incorrectly?

Thanks in advance for any hints!

The Kong ingress is provisioned using the stable/kong Helm chart.
Some relevant bits from values.yaml:

proxy:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-west-2:XXXX
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: 443
  http:
    enabled: true
    servicePort: 80
    containerPort: 8000
  tls:
    enabled: true
    servicePort: 443
    containerPort: 8443
    overrideServiceTargetPort: 8000 # <=

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: adminer
  annotations:
    kubernetes.io/ingress.class: kong
    configuration.konghq.com: ingress-api
spec:
  rules:
    - host: XXXX
      http:
        paths:
          - path: /
            backend:
              serviceName: adminer
              servicePort: 80

apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
  name: ingress-api
route:
  protocols:
    - https
  https_redirect_status_code: 302
Kong Ingress Annotations (Redirect)
#2

You need to set KONG_TRUSTED_IPS environment variable so that Kong trusts the headers sent by ELB:

#3

Hi @hbagdi @thecodingrobot

I have the same issue. Were you able to resolve it?

Ingress and KongIngress definitions:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: "2048-ingress"
  namespace: "2048-game"
  annotations:
    kubernetes.io/ingress.class: kong
    kubernetes.io/tls-acme: "true"
    configuration.konghq.com: https-only
  labels:
    app: 2048-ingress
spec:
  rules:
    - host: 2048.test.com
      http:
        paths:
          - path: /
            backend:
              serviceName: "service-2048"
              servicePort: 80
---
apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
    name: https-only
    namespace: 2048-game
route:
  protocols:
  - https
  https_redirect_status_code: 302

Both HTTP and HTTPS are redirected causing loop.

sh-4.2$ curl -I http://2048.test.com
HTTP/1.1 302 Moved Temporarily
Date: Tue, 11 Feb 2020 16:07:26 GMT
Content-Type: text/html
Content-Length: 110
Connection: keep-alive
Location: https://2048.test.com
X-Kong-Response-Latency: 0
Server: kong/1.4.3

sh-4.2$ curl -I https://2048.test.com
HTTP/1.1 302 Moved Temporarily
Date: Tue, 11 Feb 2020 16:07:33 GMT
Content-Type: text/html
Content-Length: 110
Connection: keep-alive
Location: https://2048.test.com
X-Kong-Response-Latency: 1
Server: kong/1.4.3
SSL Redirect from http to https
#4

Can you set KONG_TRUSTED_IPS=0.0.0.0/0 and try?

#5

@hbagdi, thanks for reply. I have updated the ingress-kong deployment env variables to add the KONG_TRUSTED_IPS. Is that what you mean? The result is the same, both http and https is being redirected to https causing loop.

spec:
  containers:
  - env:
    - name: KONG_DATABASE
      value: "off"
    - name: KONG_NGINX_WORKER_PROCESSES
      value: "1"
    - name: KONG_NGINX_HTTP_INCLUDE
      value: /kong/servers.conf
    - name: KONG_ADMIN_ACCESS_LOG
      value: /dev/stdout
    - name: KONG_ADMIN_ERROR_LOG
      value: /dev/stderr
    - name: KONG_ADMIN_LISTEN
      value: 127.0.0.1:8444 ssl
    - name: KONG_PROXY_LISTEN
      value: 0.0.0.0:8000, 0.0.0.0:8443 ssl http2
    - name: KONG_TRUSTED_IPS
      value: 0.0.0.0/0
#6 
KONG_TRUSTED_IPS=0.0.0.0/0,::/0

Can you change trusted ips to that?

To make sure, are you using ELB or NLB? ELB supports x-forwarded-proto and this should work with that configuration.

Kong correctly parses forwarded-proto and matches the route accordingly. I tested this locally to make sure of it.

#7

It worked! I’m using ELB. Only the http is redirected to https now, so loop issue was resolved. Thanks a lot @hbagdi.

#8

We should add this to docs somewhere. I’m not sure where though.

#9

I’ve been able to confirm this works in my use case. However, I’m wondering how this would work in the context of a Helm chart. Looking through the templates, it doesn’t seem supported.

Would it be useful if I created a PR to allow this as an env parameter?

#10

Looks like this is already handled here

#11

Hi,
What about NLB ? Is there a way to configure HTTP to HTTPS redirection with that load balancer ?
Thanks,

#12

The same approach should work if you are terminating HTTP at NLB.

#13

Hi @hbagdi,

I’m using nlb and facing The plain HTTP request was sent to HTTPS port, i tried adding KONG_TRUSTED_IPS to kong deployment but no luck.
below is my kong-proxy service

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:...
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb

  ports:
  - name: proxy
    nodePort: 31193
    port: 80
    protocol: TCP
    targetPort: 8000
  - name: proxy-ssl
    nodePort: 30102
    port: 443
    protocol: TCP
    targetPort: 8443

and my ingress be like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    konghq.com/override: https-only
    konghq.com/strip-path: "true"

and ingress-kong pod variables:

Environment:
  KONG_PROXY_LISTEN:            0.0.0.0:8000, 0.0.0.0:8443 ssl http2
  KONG_PORT_MAPS:               80:8000, 443:8443
  KONG_ADMIN_LISTEN:            127.0.0.1:8444 ssl
  KONG_STATUS_LISTEN:           0.0.0.0:8100
  KONG_DATABASE:                off
  KONG_NGINX_WORKER_PROCESSES:  2
  KONG_ADMIN_ACCESS_LOG:        /dev/stdout
  KONG_ADMIN_ERROR_LOG:         /dev/stderr
  KONG_PROXY_ERROR_LOG:         /dev/stderr
  KONG_TRUSTED_IPS:             0.0.0.0/0, ::/0

If i change proxy-ssl target port to 8000 then got a redirect loop

  - name: proxy-ssl
    nodePort: 30102
    port: 443
    protocol: TCP
    targetPort: 8000

Not sure what i’m missing here. Thank you in advance !

1 Like
#14

Hi team and @hbagdi
We have we same issue on our AWS NLB. We use overrideServiceTargetPort to resolve “The plain HTTP request was sent to HTTPS port”.
If the use

konghq.com/protocols: "https"
konghq.com/https-redirect-status-code: "301"

it makes too many redirections(loop) and this looks impossible now to configure http-https redirection with overrideServiceTargetPort

1 Like
#15

Hello Team,

I have had the same issue. I am using AWS NLB. If we point the target group of port 443 to HTTPS port of KIC (Kong Ingress Controller), we will see an error message “The plain HTTP request was sent to HTTPS port”. However, If we point the target group of port 443 to the HTTP port of KIC, everything is worked fine, but we can not redirect HTTP to HTTPS because the x-forwarding-port is always HTTP. It makes too many redirections

#16

it’s because you send unencrypted data to a SSL encrypted port, try adjusting your target port to http port will help, or switch to ELB which is easier.

#17

Any help here would be appreciated. I have exactly the same problem as the guys above.

If we do TLS on a NLB, we need to set the ports for the service in kong-proxy to this

spec:
  ports:
    - name: proxy
      port: 80
      protocol: TCP
      targetPort: 8000
    - name: proxy-ssl
      port: 443
      protocol: TCP
      targetPort: 8000

Otherwise we’re getting The plain HTTP request was sent to HTTPS port

On our ingress I put

  annotations:
    kubernetes.io/ingress.class: kong
    konghq.com/protocols: "https"
    konghq.com/https-redirect-status-code: "301"

But this causes an endless redirect loop

This is deployment with the environment variables

containers:
        - env:
            - name: KONG_PROXY_LISTEN
              value: 0.0.0.0:8000, 0.0.0.0:8443 ssl http2
            - name: KONG_PORT_MAPS
              value: 80:8000, 443:8443
            - name: KONG_ADMIN_LISTEN
              value: 127.0.0.1:8444 ssl
            - name: KONG_STATUS_LISTEN
              value: 0.0.0.0:8100
            - name: KONG_DATABASE
              value: "off"
            - name: KONG_NGINX_WORKER_PROCESSES
              value: "2"
            - name: KONG_ADMIN_ACCESS_LOG
              value: /dev/stdout
            - name: KONG_ADMIN_ERROR_LOG
              value: /dev/stderr
            - name: KONG_PROXY_ERROR_LOG
              value: /dev/stderr
            - name: KONG_TRUSTED_IPS
              value: 0.0.0.0/0,::/0
          image: kong:2.5

@hbagdi

1 Like
#18

Hello, after having encountered the same problem as you ‘too many redirects’ I debugged and found a solution. The redirect problem is caused by this part of the conf:

  tls:
# Use it for ELB load balancer type not NLB
    overrideServiceTargetPort: 8000

so I removed this overrideServiceTargetPort which avoids the rewriting of port 8443 on port 8000.

In addition you have to use this annotation:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: “ssl”
not
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: “tcp”
at the risk of encountering a plain text error on an HTTPS connection.

Moreover, the kong doc clearly indicates that for an L4 load balancer like NLB you have to use its var env:

  trusted_ips: "0.0.0.0/0,::/0"
  proxy_listen: "0.0.0.0:8000 proxy_protocol, 0.0.0.0:8443 ssl proxy_protocol"
  real_ip_header: "proxy_protocol"

But this is not enough, since the proxy listens on port 8443 with the SSL protocol, the healthcheck executed by the load balancer uses the TCP protocol which generates an error of the type:

"broken header: "" while reading PROXY protocol".

The solution I found to this is to add a var env:

status_listen: "0.0.0.0:8100"

you have to add this annotation too:

service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "8100"

This allows the load balancer to get the status on port 8100 via TCP without generating an error.
hoping to have helped you !

#19

Hi @Enzos23 your method didn’t work for me . It returns

* TLSv1.2 (IN), TLS alert, close notify (256):
* Empty reply from server
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
curl: (52) Empty reply from server

ELB with TLS is not able to direct my request to Kong when i remove the overrideServiceTargetPort

#20

I too am getting empty reply with the solution posted from @Enzos23 . I also get a redirect loop if I try to enforce https redirect. I am using a NLB with the following configurations.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ingress-kong
  name: ingress-kong
  namespace: kong
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-kong
  template:
    metadata:
      annotations:
        kuma.io/gateway: enabled
        traffic.sidecar.istio.io/includeInboundPorts: ""
      labels:
        app: ingress-kong
    spec:
      containers:
      - env:
        - name: KONG_PROXY_LISTEN
          value: 0.0.0.0:8000 proxy_protocol, 0.0.0.0:8443 ssl proxy_protocol
        - name: KONG_PORT_MAPS
          value: 80:8000, 443:8443
        - name: KONG_ADMIN_LISTEN
          value: 127.0.0.1:8444 ssl
        - name: KONG_STATUS_LISTEN
          value: 0.0.0.0:8100
        - name: KONG_DATABASE
          value: "off"
        - name: KONG_NGINX_WORKER_PROCESSES
          value: "2"
        - name: KONG_KIC
          value: "on"
        - name: KONG_ADMIN_ACCESS_LOG
          value: /dev/stdout
        - name: KONG_ADMIN_ERROR_LOG
          value: /dev/stderr
        - name: KONG_PROXY_ERROR_LOG
          value: /dev/stderr
        - name: KONG_PLUGINS
          value: bundled,custom-auth
        - name: KONG_LUA_PACKAGE_PATH
          value: /opt/?.lua;;
        - name: KONG_TRUSTED_IPS
          value: 0.0.0.0/0,::/0
        - name: KONG_REAL_IP_HEADER
          value: proxy_protocol

Service

apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: xxxx
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "8100"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: xxxxx
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  name: kong-proxy
  namespace: kong
spec:
  ports:
  - name: proxy
    port: 80
    protocol: TCP
    targetPort: 8000
  - name: proxy-ssl
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    app: ingress-kong
  type: LoadBalancer