TCPIngress configuration

Kubernetes
1

Hi all

I tried the following config to expose an Oracle DB as a TCP service. The database works as I could test it with port forwarding. But, the service is not accessible from the outside. What do I miss in the config?

Thanks!

apiVersion: configuration.konghq.com/v1beta1
kind: TCPIngress
metadata:
  name: oracle19-ingress
  annotations:
    kubernetes.io/ingress.class: kong
spec:
  rules:
    - port: 25432
      backend:
        serviceName: oracle19
        servicePort: 1521

---
apiVersion: v1
kind: Service
metadata:
  name: oracle19
  labels:
    app: oracle19
spec:
  ports:
    - name: db
      protocol: TCP
      port: 1521
      targetPort: 1521
  selector:
    app: oracle19

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: oracle19
spec:
  serviceName: oracle19
  selector:
    matchLabels:
      app: oracle19
  template:
    metadata:
      labels:
        app: oracle19
    spec:
      containers:
        - name: oracle19
          image: a-company-oracle/database:19-ee
          ports:
            - containerPort: 1521
          env:
            - name: ORACLE_PWD
              value: oracle
          volumeMounts:
            - mountPath: /opt/oracle/oradata
              name: data
      volumes:
        - name: data
          hostPath:
            path: /opt/oracle/oradata
            type: DirectoryOrCreate
2

If you port-forward to the Kong admin API and call GET /routes, do you see a oracle19-ingress route present? If not, check your controller logs to see if there’s some issue pushing configuration.

If the route is present, do you see any error logs in the proxy log? Can you show what happens when you make a test connection?

AFAIK logging for TCP routes is fairly limited, but you can attach a tcpdump sidecar to the proxy Pod and check traffic for this with tcpdump -npi any port 25432 or port 1521 to see if there are any issues establishing connections on either end.

3

Are you expose tcp port in ingress pod and ingress service ?

4

Thank you very much for your replies! As far as I understood the configuration settings the ports are exposed. I hope that my configuration posted above is correct. If there is any mistake please let me know. Currently, the JDBC client claims: “Connection refused”.

What I checked so far

  • The nftables rules are ok (checked with positive / negative testing)
  • The /routes returns oracle-19.ingress.0
  • I installed the sidecar and I am generally able to get information on some ports, let’s say admin 8001 etc. However, there is no traffic on port 1521 and 25432.

Here is the info about the ingress from k8s:

apiVersion: configuration.konghq.com/v1beta1
kind: TCPIngress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"configuration.konghq.com/v1beta1","kind":"TCPIngress","metadata":{"annotations":{"kubernetes.io/ingress.class":"kong"},"name":"oracle19-ingress","namespace":"default"},"spec":{"rules":[{"backend":{"serviceName":"oracle19","servicePort":1521},"port":25432}]}}
    kubernetes.io/ingress.class: kong
  creationTimestamp: '2022-04-03T08:20:36Z'
  generation: 1
  managedFields:
    - apiVersion: configuration.konghq.com/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
            f:kubernetes.io/ingress.class: {}
        f:spec:
          .: {}
          f:rules: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: '2022-04-03T08:20:36Z'
    - apiVersion: configuration.konghq.com/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:loadBalancer:
            .: {}
            f:ingress: {}
      manager: manager
      operation: Update
      subresource: status
      time: '2022-04-03T08:20:37Z'
  name: oracle19-ingress
  namespace: default
  resourceVersion: '879452'
  uid: 41d8b97a-001e-4e75-a809-c0a6c337038d
  selfLink: >-
    /apis/configuration.konghq.com/v1beta1/namespaces/default/tcpingresses/oracle19-ingress
status:
  loadBalancer:
    ingress:
      - ip: xxx.xxx.xxx.xxx
spec:
  rules:
    - backend:
        serviceName: oracle19
        servicePort: 1521
      port: 25432

Thanks and have a great day.

5

The external client getting refused indicates that the proxy isn’t actually listening for it. You’d need to add 25432 as a TCP stream listen

When using the chart, this is handled via proxy.stream entries.

6

Thanks for pointing me to the relevant information. Based on the docs and values I ended up with the following config that however throws an error. As far as I can see it’s configured correctly? At least the externalTrafficPolicy works.

proxy:
  externalTrafficPolicy: Local
  stream:
    - protocol: TCP
      containerPort: 1521
      servicePort: 1521
      hostPort: 25432

Error message:

coalesce.go:220: warning: cannot overwrite table with non table for kong.proxy.stream (map[])
c
7

Check your Helm version’s up to date? Pasting that stream section into one of the test values.yamls and rendering it worked fine for me, and search results for that error indicate there was a bug in earlier Helm versions that would result in it being thrown spuriously.

8

Thank you very much for your inputs. Here is what I’ve found:

~ » helm search repo kong/kong --versions
NAME     	CHART VERSION	APP VERSION	DESCRIPTION
kong/kong	2.7.0        	2.7        	The Cloud-Native Ingress and API-management

What I’ve currently installed:

~ » helm list --all-namespaces
NAME	NAMESPACE	REVISION	UPDATED                                 	STATUS  	CHART     	APP VERSION
kong	gateway  	16      	2022-04-08 22:36:14.003648275 +0200 CEST	deployed	kong-2.7.0	2.7

So if there is a new version I wouldn’t know which version it could be.

9

Do you get the same error when rendering the values.yaml I uploaded? If not, can you share a complete values.yaml that results in that error?

10

Yes, I get exactly the same error with the configuration that you provided (thanks!). When I comment out the “stream:” section, the error is gone.

11

Sorry, lost track a bit between responses: the bug is in Helm itself, helm version is what you want to check. I have:

version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.8"}

Nested `null` values do not remove keys from sub-charts · Issue #9136 · helm/helm · GitHub is the latest version (still not fixed) version of that bug, but that variant should only affect cases where you’re using the Kong chart as a subchart. It links to some of the older fixes for similar issues.

12

It seems that I use exactly the same version (beside go):

~ » helm version
version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.5"}

I also tried version 3.8.2 on debian (as well with the go1.17.5 version). May be that’s the issue.

13

Not sure then–not sure what would cause you to get the error with the same values.yaml that works for me on the same version. At this point I’d recommend filing a bug on Issues · helm/helm · GitHub since they know the internals better, and can hopefully suggest what else may be the cause of the different render results we’re seeing.