AWS ELB offload wildcard certificate with Kong Ingress Controller + Websocket


#1

Hi,

I have couple requirements need to meet:

  • Need to use existed wildcard certificate stored in AWS Certificate Manager, accomplished with AWS ELB (not really care L4 or L7). The certificate was generated from AWS Certificate Manager, therefore no chance to dump its private key.
  • Need to serve HTTPS and websocket (wss) traffic at the same time.
  • Basic rewrite to K8S service root level required. E.g.
    HTTPS example:
    [External] https://api0.demo.acme.com/httpbin/whatever -> [Internal, httpbin service] http://localhost:8000/whatever
    Websocket example:
    [External] wss://stream.acme.com/marketdata/whatever -> [Internal, websocket service] ws://localhost:80/whatever

kubectl apply -f https://bit.ly/kong-ingress

cat <<EOF| k apply -f -
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: REDACTED
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: '*'
  name: kong-proxy
  namespace: kong
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: kong-proxy
    nodePort: 31245
    port: 80
    protocol: TCP
    targetPort: 8000
  - name: kong-proxy-ssl
    nodePort: 32361
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    app: kong
  sessionAffinity: None
  type: LoadBalancer

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    configuration.konghq.com: default
  name: default
spec:
  rules:
  - host: api0.demo.acme.com
    http:
      paths:
      - backend:
          serviceName: httpbin
          servicePort: 8000
        path: /httpbin
  - host: stream.demo.acme.com
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 80
        path: /marketdata

---

apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
  name: default
proxy:
  path: /
route:
  strip_path: true
EOF

But when I curl it, it keeps return 400 Bad Request

$ curl https://api0.demo.acme.com/httpbin/ -vvvv
*   Trying 1.2.3.4...
* TCP_NODELAY set
* Connected to api0.demo.acme.com (1.2.3.4) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.demo.acme.com
*  start date: Dec  7 00:00:00 2018 GMT
*  expire date: Jan  7 12:00:00 2020 GMT
*  subjectAltName: host "api0.demo.acme.com" matched cert's "*.demo.acme.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET /httpbin/ HTTP/1.1
> Host: api0.demo.acme.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Date: Wed, 12 Dec 2018 13:23:56 GMT
< Content-Type: text/plain; charset=UTF-8
< Content-Length: 12
< Connection: close
< Server: kong/0.14.1
<
Bad request
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):

What I do wrong? Please advice. Thanks.

Also the K8S version I used is v1.10.6 (KOPS stable). K8S API server on the master has following admission control enabled:

- Initializers
- NamespaceLifecycle
- LimitRanger
- ServiceAccount
- PersistentVolumeLabel
- DefaultStorageClass
- DefaultTolerationSeconds
- NodeRestriction
- ResourceQuota
- AlwaysPullImages
- DenyEscalatingExec

#2

Are the ports and protocol getting mixed up?

client -> AWS ELB is HTTPS but what is the protocol being used between AWS ELB -> Kong?


#3

@hbagdi

Thanks for reply.

Originally I thought it should be TCP since have successful case in k8s nginx-ingress-controller. What do you recommend?

// M


#4

Is it possible that HTTP and HTTPS are getting mixed up between the ELB and Kong?


#5

Were you able to get this working?, I’ve been trying without success, the ELB doesn’t even pick the certificate, I get an ELB with 2 TCP, while it works there is no cert assigned.


[Solved] Kong ingress controller + Aws certificate manager?
#6

@kainlite Try this

cat <<EOF| k apply -f -
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: REDACTED
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: '*'
  name: kong-proxy
  namespace: kong
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: kong-proxy-ssl
    nodePort: 32361
    port: 443
    protocol: TCP
    targetPort: 8000
  selector:
    app: kong
  sessionAffinity: None
  type: LoadBalancer
EOF

Although I switched to other vendor provided ingress already.

// M


#7

Awesome, Thank you @zanhsieh, that annotation in the kube-proxy did the trick, I’m now struggling with the ELB to serve http and https, this is really confusing.