HTTPS encryption s not enable with kong-ingress and cert-manager

Kubernetes
1

I am using kong-ingress-controller and cert-manager on my kubernetes cluster.
Currently, I want to enable the https to encrypt certificates using letsencrypt like CA

I have this problem:

I0315 12:10:56.109919       1 controller.go:168] certificates controller: Finished processing work item "default/letsencrypt-staging"
I0315 12:10:56.910004       1 logger.go:73] Calling GetAuthorization
I0315 12:10:57.135213       1 logger.go:93] Calling HTTP01ChallengeResponse
I0315 12:10:57.150924       1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.151110       1 controller.go:162] certificates controller: syncing item 'default/letsencrypt-staging'
I0315 12:10:57.151113       1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
I0315 12:10:57.151521       1 sync.go:274] Need to create 1 challenges
I0315 12:10:57.151891       1 issue.go:160] Order default/letsencrypt-staging-2613163196 is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate.
I0315 12:10:57.152022       1 controller.go:168] certificates controller: Finished processing work item "default/letsencrypt-staging"
I0315 12:10:57.160718       1 sync.go:323] Waiting for all challenges for order "letsencrypt-staging-2613163196" to enter 'valid' state
I0315 12:10:57.160804       1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.160870       1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
I0315 12:10:57.160739       1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I0315 12:10:57.161051       1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 12:10:57.161214       1 sync.go:274] Need to create 0 challenges
I0315 12:10:57.161263       1 sync.go:323] Waiting for all challenges for order "letsencrypt-staging-2613163196" to enter 'valid' state
I0315 12:10:57.161287       1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.562383       1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
I0315 12:10:57.562738       1 sync.go:274] Need to create 0 challenges
I0315 12:10:57.562755       1 sync.go:323] Waiting for all challenges for order "letsencrypt-staging-2613163196" to enter 'valid' state
I0315 12:10:57.562906       1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.563018       1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I0315 12:10:57.563231       1 logger.go:68] Calling GetChallenge
I0315 12:10:57.833421       1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 12:10:57.833648       1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
I0315 12:10:57.833755       1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I0315 12:10:57.833820       1 sync.go:274] Need to create 0 challenges
I0315 12:10:57.833828       1 sync.go:323] Waiting for all challenges for order "letsencrypt-staging-2613163196" to enter 'valid' state
I0315 12:10:57.833838       1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.833858       1 logger.go:103] Calling Discover
I0315 12:10:57.856136       1 pod.go:64] No existing HTTP01 challenge solver pod found for Certificate "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:57.923080       1 service.go:51] No existing HTTP01 challenge solver service found for Certificate "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:57.989596       1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=4095675862,certmanager.k8s.io/acme-http-token=657526223
I0315 12:10:57.989682       1 ingress.go:98] No existing HTTP01 challenge solver ingress found for Challenge "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:58.014803       1 controller.go:178] ingress-shim controller: syncing item 'default/cm-acme-http-solver-jr4fg'
I0315 12:10:58.014842       1 sync.go:64] Not syncing ingress default/cm-acme-http-solver-jr4fg as it does not contain necessary annotations
I0315 12:10:58.014846       1 controller.go:184] ingress-shim controller: Finished processing work item "default/cm-acme-http-solver-jr4fg"
I0315 12:10:58.015447       1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=4095675862,certmanager.k8s.io/acme-http-token=657526223
I0315 12:10:58.033431       1 sync.go:173] propagation check failed: wrong status code '404', expected '200'
I0315 12:10:58.079504       1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 12:10:58.079616       1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I0315 12:10:58.079569       1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'

get this message No existing HTTP01 challenge solver pod found for Certificate "default/letsencrypt-staging-2613163196-0"

Currently, my certificate in staging environment has not been validated by letsencrypt.
I’ve posted this inconvenient in letsencrypt community

I have been checking the logs of my kong ingress controller pod and I get this output:

kubectl logs -n kong kong-ingress-controller-667b4748d4-ccj8z -c ingress-controller

I0318 09:47:04.293414       6 controller.go:128] syncing Ingress configuration...
I0318 09:47:04.556011       6 kong.go:1075] cert: 0xc000812d70
I0318 09:47:04.995787       6 kong.go:113] syncing global plugins
W0318 09:47:05.248513       6 kong.go:335] there is no custom Ingress configuration for rule default/kong-ingress-zcrm365
W0318 09:47:05.332825       6 kong.go:751] there is no custom Ingress configuration for rule default/kong-ingress-zcrm365
I0318 09:47:09.067196       6 controller.go:128] syncing Ingress configuration...
I0318 09:47:10.339943       6 kong.go:1075] cert: 0xc00086b670
I0318 09:47:10.593221       6 kong.go:113] syncing global plugins
W0318 09:47:10.777719       6 kong.go:335] there is no custom Ingress configuration for rule default/kong-ingress-zcrm365
W0318 09:47:10.860244       6 kong.go:751] there is no custom Ingress configuration for rule default/kong-ingress-zcrm365
I0318 09:47:50.066760       6 controller.go:128] syncing Ingress configuration...
I0318 09:47:51.282199       6 kong.go:1075] cert: 0xc0007f9d30
I0318 09:47:51.569891       6 kong.go:113] syncing global plugins
W0318 09:47:51.796106       6 kong.go:335] there is no custom Ingress configuration for rule default/kong-ingress-zcrm365
W0318 09:47:51.886539       6 kong.go:751] there is no custom Ingress configuration for rule default/kong-ingress-zcrm365
[I]

Is necessary to create a custom Ingress configuration (maybe some additional customized annotation) in order to reference the kong-ingress-controller?

I have been exploring some cert-manager issues and I found this https://github.com/Kong/kubernetes-ingress-controller/issues/162 in which even @hbagdi have some participation.

I think that the problem may be in the way of how cert-manager could be work or not with kong. cert-manager also creates a new Ingress resource to handle the ACME http01 validation named cm-acme-http-solver-qxtdg

I’ve added to my ingress resource the kong ingress class and my ingress has stayed of this way:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kong-ingress-zcrm365
  #namespace: default
  annotations:
    # kubernetes.io/ingress.class: "nginx" # No add it, this remove to kong-ingress-controller kong
    kubernetes.io/ingress.class: "kong"
    certmanager.k8s.io/acme-challenge-type: http01
    # configuration.konghq.com: global-kong-ingress-rules

    certmanager.k8s.io/cluster-issuer: letsencrypt-staging
spec:
  rules:
  - host: test1kongletsencrypt.possibilit.nl
    http:
      paths:
        - path: "/"
          backend:
            serviceName: zcrm365dev
            servicePort: 80
  tls: 
  - hosts:
    - test1kongletsencrypt.possibilit.nl
    secretName:  letsencrypt-staging

And the new Ingress resource to handle the ACME http01 validation that cert-manager create are using kong … or un least that seem …

⟩ kubectl get ingress cm-acme-http-solver-x9kns -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: kong # LOOK HERE
    nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0
  creationTimestamp: "2019-03-18T10:20:50Z"
  generateName: cm-acme-http-solver-
  generation: 1
  labels:
    certmanager.k8s.io/acme-http-domain: "xxxxxx"
    certmanager.k8s.io/acme-http-token: "xxxxx"
  name: cm-acme-http-solver-x9kns
  namespace: default
  ownerReferences:
  - apiVersion: certmanager.k8s.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Challenge
    name: letsencrypt-staging-710862264-0
    uid: 809be9f7-4967-11e9-a113-e27267a7d354
  resourceVersion: "1159494"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/cm-acme-http-solver-x9kns
  uid: 812950a0-4967-11e9-a113-e27267a7d354
spec:
  rules:
  - host: test1kongletsencrypt.possibilit.nl
    http:
      paths:
      - backend:
          serviceName: cm-acme-http-solver-kbd9p
          servicePort: 8089
        path: /.well-known/acme-challenge/xxxxxxxxx
status:
  loadBalancer:
    ingress:
    - ip: 52.166.60.158

The https://github.com/jetstack/cert-manager/issues/958 issue was the first source through I can reference the other issues.

Is possible that kong and cert-manager do not work together to enable https with several CA like letsencrypt?

2

I’ve applied this acme-kong-kube-helper github.com/ollystephens/acme-kong-kube-helper in order to solve the http01 validation problem in staging environment, and all it’s works.

When the http01 validation to be performed, is necessary use the letsencrypt production environment to get the https encryption.

The problem is mainly that cert-manager have some problems to work with other ingress controller different to nginx. This helper is a temporal solution. Currently cert-manager work in this feature to solve this https://github.com/jetstack/cert-manager/issues/1097