Traffic to kong-ingress from outside the K8s with nginx-ingress and multiple hosts/ingresses

Kubernetes
1

I have K8s cluster on bare machines (Rancher), with ingress-nginx from Rancher.
I deployed kong with official chart (1.1.1), with “proxy.type: ClusterIP” (I left the other settings by default).

So, from outside of cluster I have to redirect traffic to kong-ingress by ingress-nginx.
I created in kong namespace ingress, which is set to service kong-kong-proxy

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echo2-frontend-rke-to-kong
  namespace: kong
spec:
  rules:
  - host: my.host.name
    http:
      paths:
      - backend:
          serviceName: kong-kong-proxy
          servicePort: 80
  tls: # < placing a host in the TLS config will indicate a cert should be created
  - hosts:
    - my.host.name
    secretName: echo2-frontend-rke-to-kong-tls-cert

In different namespace I created kong-ingres, with plugins etc (and with echo service for testing purpose):
apiVersion: extensions/v1beta1

kind: Ingress
metadata:
  name: echo2-ingress-kong
  annotations:
    kubernetes.io/ingress.class: "kong"
    plugins.konghq.com: acl,jwt
spec:
  rules:
  - host: my.host.name
    http:
      paths:
      - backend:
          serviceName: echo
          servicePort: 80

This works fine (I can curl https://my.host.name, JWT authentication + ACL works).

When I tried to add second host (ingress-nginx + kong-ingress as above, only with different host and secretName), I get immediately “Request Header Or Cookie Too Large” (for both hosts), and in kong proxy logs:

2020/02/04 12:20:24 [warn] 24#0: *42315 [lua] reports.lua:73: log(): [reports] unknown request scheme: http while logging request, client: 10.42.4.65, server: kong, request: “GET / HTTP/1.1”, host: “my.host.name”
10.42.4.65 - - [04/Feb/2020:12:20:24 +0000] “GET / HTTP/1.1” 494 46 “-” “curl/7.67.0”
(…)
10.42.4.65 - - [04/Feb/2020:12:20:24 +0000] “GET / HTTP/1.1” 494 46 “-” “curl/7.67.0”
10.42.5.1 - - [04/Feb/2020:12:20:24 +0000] “GET / HTTP/1.1” 494 46 “-” “curl/7.67.0”

Is any way to use many hosts/ingresses in this scenario (traffic from outsite of cluster by ingress-nginx to one kong-ingress deployment?)

2

This seems like you need larger headers being sent with the request.

You need to increase the header buffer for Nginx in ingress-nginx and kong both:

3

I don’t think is the case. Seems kong proxy works badly with service type ClusterIP.
So scenario “ingress-nginx -> kong proxy service (type ClusterIP) - kong-ingress” with default configs seems to be unreliable.
For example - I created only one ingress-nginx to kong proxy service.
Until I don’t configured proper kong-ingress for the same host as in ingress-nginx (with set kubernetes.io/ingress.class: “kong”), I have with curl “Request Header Or Cookie Too Large” and a lot 494 errors in kong proxy pod.
When I added proper kong-ingress instance, errors disappeared.
Whan I add second kong-ingress (without ingress-nginx), with different host errors for first ingress returned.

So I twisted to NodePort.
cheers

4

Everything working just fine with ClusterIP service, when you change default kong ingress class name to another value.

Like wrote @hbagdi on github:
" That’s a special behavior of the default kong ingress class where any ingress resource that is not annotated is picked up.
If you change the ingress class to anything else, let’s say, kong-foo , then the controller will not pick up ingress resources without any ingress class."

That was the reason for above failure! Remember, you have to use your new ingress class name on every Kong CRD object (plugin, consumer etc).