Ingress Controller registers HTTPS services as HTTP (and port 443 as port 80)

#1

When creating HTTPS Kong Services and Routes via the Ingress Controller, my services are always defaulted to protocol=HTTP and port=80 in postgres.
However, the hostname is correctly configured as namespace.example-api.https.
Once I run this update in postgres, HTTPS works:
UPDATE services SET port = 80, protocol = 'https'

This is quite problematic for my use case, as we MUST use e2e TLS encryption, and any deployment requires a manual update to the postgres service records. This also has to be run after any k8 deployment to an existing k8 deployment resource, which leads me to believe that this is either related to certificates/TLS, or (more likely) kong’s parsing of ingress resources’ https configuration.
I’ve also tried using 443 rather than https as the value for servicePort, to no avail.

Here’s an example ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-ingress
  namespace: example
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: "kong"
    configuration.konghq.com: example-kongingress
spec:
  tls:
  - hosts:
    - "example.ingress.com"
    secretName: example-cert
  rules:  
    - host: "example.ingress.com"
      http:
        paths:
          - path: "/v1/example"
            backend:
              serviceName: example-svc
              servicePort: https 
---
apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
  name:  example-kongingress
  namespace: example
route:
  methods:
  - POST
  - PUT
  - DELETE
  - PATCH
  - GET
  - OPTIONS
  protocols:
  - https
  - http
  strip_path: true
  preserve_host: true
---

Here’s the ingress controller log for this event (which doesn’t tell us much about what’s going on with the https configuration):

Event(v1.ObjectReference{Kind:"Ingress", Namespace:"example", Name:"example-ingress", UID:"some-uuid", APIVersion:"extensions", ResourceVersion:"2549274", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress example/example-ingress

and the resulting postgres service record:

I’ve dug into the go src for the kong ingress controller, and have found that this issue is probably occurring within
kubernetes-ingress-controller/internal/ingress/controller/store/store.go

	ingEventHandler := cache.ResourceEventHandlerFuncs{
		AddFunc: func(obj interface{}) {
			ing := obj.(*extensions.Ingress)
			if !class.IsValid(&ing.ObjectMeta) {
				a, _ := parser.GetStringAnnotation(class.IngressKey, &ing.ObjectMeta)
				glog.Infof("ignoring add for ingress %v based on annotation %v with value %v", ing.Name, class.IngressKey, a)
				return
			}

			recorder.Eventf(ing, corev1.EventTypeNormal, "CREATE", fmt.Sprintf("Ingress %s/%s", ing.Namespace, ing.Name))

			recorder.Eventf(ing, corev1.EventTypeNormal, "CREATE", fmt.Sprintf("Ingress %s/%s", ing.Namespace, ing.Name))

			updateCh.In() <- Event{
				Type: CreateEvent,
				Obj:  obj,
			}
		},
(skipping some irrelevant lines here...)

	epEventHandler := cache.ResourceEventHandlerFuncs{
		AddFunc: func(obj interface{}) {
			updateCh.In() <- Event{
				Type: CreateEvent,
				Obj:  obj,
			}
		},
		DeleteFunc: func(obj interface{}) {
			updateCh.In() <- Event{
				Type: DeleteEvent,
				Obj:  obj,
			}
		},
		UpdateFunc: func(old, cur interface{}) {
			oep := old.(*corev1.Endpoints)
			ocur := cur.(*corev1.Endpoints)
			if !reflect.DeepEqual(ocur.Subsets, oep.Subsets) {
				updateCh.In() <- Event{
					Type: UpdateEvent,
					Obj:  cur,
				}
			}
		},
	}

	serviceEventHandler := cache.ResourceEventHandlerFuncs{
		UpdateFunc: func(old, cur interface{}) {
			glog.Infof("UPDATING SERVICE, OBJECT IS:%+v\n", cur)
			updateCh.In() <- Event{
				Type: ConfigurationEvent,
				Obj:  cur,
			}
		},
		DeleteFunc: func(obj interface{}) {
			updateCh.In() <- Event{
				Type: DeleteEvent,
				Obj:  obj,
			}
		},
	}

I can see that we’re sending events to a go channel, and somewhere on the other end, that channel is being listened to, and updating postgres accordingly.
I can see that there is an add endpoint function, but only an update service function, and I believe that kong uses endpoints rather services to proxy requests to, but I’m a bit shaky on how that works.

I’m looking into rebuilding in the ingress controller image from scratch, and including better logging to view the data prior to being written to postgres.

In lieu of that, has anyone else run into this problem and found the root of the issue?

Thanks!

  • Dylan
0 Likes

#2

Surely I’m not the only one that’s run into this issue, or maybe people aren’t serving https to their back end services / pods…

0 Likes

#3

Could you try doing the following?

Instead of the example-kongingress you use, use the following one:

apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
  name:  example-kongingress
  namespace: example
route:
  methods:
  - POST
  - PUT
  - DELETE
  - PATCH
  - GET
  - OPTIONS
  protocols:
  - https
  - http
  strip_path: true
  preserve_host: true
proxy:
  protocol: https

In the ingress spec, there is no way to determine if the upstream connection should be HTTP or HTTPS (there might be a way to do it by inspecting the service definitions but we don’t use that currently).
Setting proxy.protocol in KongIngress will direct Kong to use TLS while talking the to service upstream, achieve the E2E TLS you require.

0 Likes

#4

Thanks for replying @hbagdi!

So, according to the documentation,
*Note:* Is not required to define the complete object, one can define only one of the `upstream` , `proxy` or `route` sections
so my understanding was that we could not specify proxy information as well as route information.

However, I have also tried using both, like you suggested (prior to initially posting this issue) to no avail.

I’m wondering if the kubernetes api isn’t actually parsing the values from the manifest correctly and is sending http and 80 to the ingress controller and then the kong api, but I’m guessing it’s more closely related to the ingress controller. Not sure though, of course :wink:

I have also tried rebuilding the ingress controller image using the source code, and modifying the default values to https and 443 in kubernetes-ingress-controller/internal/ingress/controller/kong.golike so:

			name := buildName(backend, location)
			for _, upstream := range ingressCfg.Backends {
				if upstream.Name != backend {
					continue
				}

				// defaults
				proto := "https"
				port := 443

				s, err := client.Services.Get(nil, &name)
				if kong.IsNotFoundErr(err) {
					s = &kong.Service{
						Name:     kong.String(name),
						Path:     kong.String("/"),
						Protocol: kong.String(proto),
						Host:     kong.String(name),
						Port:     kong.Int(port),
						Retries:  kong.Int(5),
					}
				}

but when building the image, the go compiler complains that:

make ARCH=amd64 build container
make[1]: Entering directory '/home/dbaker/go/kubernetes-ingress-controller'
docker rmi -f gcr.io/development-229523/kong-ingress-controller-amd64:0.3.0 || true
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -installsuffix cgo \
    -ldflags "-s -w -X github.com/kong/kubernetes-ingress-controller/version.RELEASE=0.3.0 -X github.com/kong/kubernetes-ingress-controller/version.COMMIT=git-cc8ccdb -X github.com/kong/kubernetes-ingress-controller/version.REPO=https://github.com/Kong/kubernetes-ingress-controller.git" -gcflags=-trimpath=/home/dbaker/go/kubernetes-ingress-controller -asmflags=-trimpath=/home/dbaker/go/kubernetes-ingress-controller \
    -o /tmp/tmp.H2iaePOefM/rootfs/kong-ingress-controller github.com/kong/kubernetes-ingress-controller/cli/ingress-controller
# github.com/kong/kubernetes-ingress-controller/internal/ingress/status
../src/github.com/kong/kubernetes-ingress-controller/internal/ingress/status/status.go:105:15: not enough arguments in call to s.elector.Run
    have ()
    want (context.Context)
../src/github.com/kong/kubernetes-ingress-controller/internal/ingress/status/status.go:201:3: cannot use func literal (type func(<-chan struct {})) as type func(context.Context) in field value
Makefile:146: recipe for target 'build' failed
make[1]: *** [build] Error 2
make[1]: Leaving directory '/home/dbaker/go/kubernetes-ingress-controller'
Makefile:85: recipe for target 'sub-container-amd64' failed
make: *** [sub-container-amd64] Error 2

I’ve attempted to remedy this, but am somewhat new to go, and have not had much luck so far.
If I can just rebuild this image, I believe I will have solved this problem.

Any advice on rebuilding?

Thanks in advance!

  • Dylan
0 Likes

#5

I see how it can be interpreted the other way, we will update the doc (PR welcome as well).

You shouldn’t need to rebuild the Ingress Controller.
One point that I missed in my previous reply (my apologies) is that you need to add the configuration.konghq.com annotation on the k8s Service, to which you want Kong to talk over TLS. Please try this and things will work.

0 Likes