Kong 3.9.1 plugin loading issue, stuck at wait for db

,
1

Hi,

I m trying to upgrade the kong 3.7 to 3.9.1. But while upgrading it, im facing the following error
wait-for-db

Error: /usr/local/share/lua/5.1/kong/cmd/start.lua:99: nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:7
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:708: error loading plugin schemas: on plugin 'fcjwtissuer': fcjwtissuer plugin is enabled but not installed;
no plugin found; on plugin 'pathfinder': pathfinder plugin is enabled but not installed;
no plugin found; on plugin 'appauthoriser': appauthoriser plugin is enabled but not installed;
no plugin found; on plugin 'jwtissuer': jwtissuer plugin is enabled but not installed;
no plugin found
stack traceback:
        [C]: in function 'assert'
        /usr/local/share/lua/5.1/kong/init.lua:708: in function 'init'
        init_by_lua(nginx-kong.conf:57):3: in main chunk
o plugin found; on plugin 'appauthoriser': appauthoriser plugin is enabled but not installed;
no plugin found; on plugin 'jwtissuer': jwtissuer plugin is enabled but not installed;
no plugin found
stack traceback:
        [C]: in function 'assert'
        /usr/local/share/lua/5.1/kong/init.lua:708: in function 'init'
        init_by_lua(nginx-kong.conf:57):3: in main chunk


  Run with --v (verbose) or --vv (debug) for more details

my docker file

FROM kong/kong:3.9.1

USER root

## Install custom js plugins
RUN apt-get update -y && apt install -y g++ curl
RUN curl -fsSL https://deb.nodesource.com/setup_20.x -o nodesource_setup.sh
RUN bash -E nodesource_setup.sh
RUN apt-get install nodejs -y
RUN npm install -g kong-pdk
RUN mkdir -p /usr/local/kong/js-plugins &&  chown -R kong:kong /usr/local/kong/js-plugins
COPY --chown=kong ./plugins /usr/local/kong/js-plugins

RUN rm -r /usr/local/kong/js-plugins/oidc

WORKDIR /usr/local/kong/js-plugins/oauth2saml

RUN npm install

WORKDIR /usr/local/kong/js-plugins/oauth2jwt

RUN npm install

WORKDIR /usr/local/kong/js-plugins/jwtissuer

RUN npm install

WORKDIR /usr/local/kong/js-plugins/pathfinder

RUN npm install

WORKDIR /usr/local/kong/js-plugins/appauthoriser

RUN npm install

WORKDIR /usr/local/kong/js-plugins/apiauthoriser

RUN npm install

WORKDIR /usr/local/kong/js-plugins/fcjwtissuer

RUN npm install

USER kong

js env:

  pluginserver_names: js
  pluginserver_js_socket: /usr/local/kong/js_pluginserver.sock
  pluginserver_js_start_cmd: /usr/bin/kong-js-pluginserver -v --plugins-directory /usr/local/kong/js-plugins
  pluginserver_js_query_cmd: /usr/bin/kong-js-pluginserver --plugins-directory /usr/local/kong/js-plugins --dump-all-plugins
2

Hi @Sathishkumar_Sat

I wanted to provide you with an answer from an internal tool we are testing, please let me know if the results here are helpful or not, and if not, any further information you can provide that I can search off of.

Thanks-
Rick

The error message you are seeing:

error loading plugin schemas: on plugin 'fcjwtissuer': fcjwtissuer plugin is enabled but not installed;
no plugin found; on plugin 'pathfinder': pathfinder plugin is enabled but not installed;
no plugin found; on plugin 'appauthoriser': appauthoriser plugin is enabled but not installed;
no plugin found; on plugin 'jwtissuer': jwtissuer plugin is enabled but not installed;
no plugin found

means that Kong is unable to find the source code for your custom plugins (fcjwtissuer, pathfinder, appauthoriser, jwtissuer). This is a common issue when upgrading or deploying Kong, especially with custom plugins.

Possible causes and solutions:

  1. Plugin files are missing from the expected directory:
    Kong expects the plugin code to be present in a specific directory (for JS plugins, this should be within the directory specified by --plugins-directory). Make sure your custom plugin directories and their code are present in /usr/local/kong/js-plugins inside the running container. If you are building a Docker image, ensure the files are copied into the image and not overwritten by any volume mounts at runtime.
    plugin is enabled but not installed

  2. Incorrect or missing configuration for plugin server:
    Double-check your environment variables for the JS pluginserver. The variables you provided look correct, but ensure they are actually set in your running container and that /usr/bin/kong-js-pluginserver exists and is executable.

  3. Volume mounts overwriting plugin directories:
    If you are using Kubernetes or Docker volumes, make sure you are not mounting over /usr/local/kong/js-plugins or any other directory containing your plugins, as this would hide the files you added during the image build.
    Why am I getting plugin not found error on kubernetes but not on docker when using custom image with go plugin

  4. Plugin is enabled in configuration but not present on the node:
    If the plugin is referenced in the database (from a previous node or migration), but the current node does not have the plugin installed, you will see this error. You must ensure that all Kong nodes have the custom plugin code installed and that the plugin name is included in the KONG_PLUGINS environment variable or equivalent configuration.
    On a new kong node kong fails to start with an error plugin is in use but not enabled

What you should do:

  • Verify that all your custom plugin directories and files exist in /usr/local/kong/js-plugins in the running container.
  • Ensure your Dockerfile copies the plugin code into the image and that no runtime mounts overwrite this directory.
  • Confirm that your environment variables for the JS pluginserver are set and correct.
  • Make sure the plugin names are included in the KONG_PLUGINS environment variable (e.g., KONG_PLUGINS=bundled,fcjwtissuer,pathfinder,appauthoriser,jwtissuer).

If after these checks the issue persists, it may be helpful to exec into the running container and manually verify the presence of the plugin files and the configuration.

If you need more specific troubleshooting steps, please provide details about how you are running Kong (Docker Compose, Kubernetes, etc.) and how you are building and deploying your image.

References:

3

Hello @rick,

Thanks for your reply :slight_smile:

Im working on deploying the kong with help of helm chart in Kubernetes running non root user(not allow to run as root)

I was able to run 3.7 version without any issues. I’m trying to upgrade the image to use 3.9.1.

I will attach the deployment yaml here.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kong
  namespace: dev-dna-kong
  labels:
    app.kubernetes.io/component: app
    app.kubernetes.io/instance: dev-dna-kong-gw
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kong
    app.kubernetes.io/version: '382.0'
    argocd.argoproj.io/instance: dev-dna-kong-gw
    helm.sh/chart: kong-2.49.0
    k8slens-edit-resource-version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: app
      app.kubernetes.io/instance: dev-dna-kong-gw
      app.kubernetes.io/name: kong
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: kong
        app.kubernetes.io/component: app
        app.kubernetes.io/instance: dev-dna-kong-gw
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: kong
        app.kubernetes.io/version: '382.0'
        helm.sh/chart: kong-2.49.0
        version: '382.0'
      annotations:
        argocd.argoproj.io/hook: PreSync
        kuma.io/gateway: enabled
        kuma.io/service-account-token-volume: kong-token
        traffic.sidecar.istio.io/includeInboundPorts: ''
    spec:
      volumes:
        - name: kong-prefix-dir
          emptyDir:
            sizeLimit: 256Mi
        - name: kong-tmp
          emptyDir:
            sizeLimit: 1Gi
        - name: kong-token
          projected:
            sources:
              - serviceAccountToken:
                  expirationSeconds: 3607
                  path: token
              - configMap:
                  name: kube-root-ca.crt
                  items:
                    - key: ca.crt
                      path: ca.crt
              - downwardAPI:
                  items:
                    - path: namespace
                      fieldRef:
                        apiVersion: v1
                        fieldPath: metadata.namespace
            defaultMode: 420
      initContainers:
        - name: clear-stale-pid
          image: kong:382
          command:
            - rm
            - '-vrf'
            - $KONG_PREFIX/pids
          env:
            - name: KONG_ADMIN_ACCESS_LOG
              value: /dev/stdout
            - name: KONG_ADMIN_ERROR_LOG
              value: /dev/stderr
            - name: KONG_ADMIN_GUI_ACCESS_LOG
              value: /dev/stdout
            - name: KONG_ADMIN_GUI_ERROR_LOG
              value: /dev/stderr
            - name: KONG_ADMIN_LISTEN
              value: 0.0.0.0:8001, [::]:8001
            - name: KONG_CLUSTER_LISTEN
              value: 'off'
            - name: KONG_DATABASE
              value: postgres
            - name: KONG_LOG_LEVEL
              value: debug
            - name: KONG_LUA_PACKAGE_PATH
              value: /opt/?.lua;/opt/?/init.lua;;
            - name: KONG_NGINX_PROXY_LARGE_CLIENT_HEADER_BUFFERS
              value: 8 24k
            - name: KONG_NGINX_WORKER_PROCESSES
              value: '5'
            - name: KONG_ONEAPI_GATEWAY_KEY
              value: 
            - name: KONG_ONEAPI_GATEWAY_SECRET
              value: 
            - name: KONG_ONEAPI_HTTPS_PROXY
              value: http://:3128
            - name: KONG_PG_DATABASE
              value: kong
            - name: KONG_PG_HOST
              value: kong-db
            - name: KONG_PG_PASSWORD
              value: 
            - name: KONG_PG_PORT
              value: '64000'
            - name: KONG_PG_SCHEMA
              value: public
            - name: KONG_PG_SSL
              value: 'off'
            - name: KONG_PG_SSL_VERIFY
              value: 'off'
            - name: KONG_PG_USER
              value: kong
            - name: KONG_PLUGINS
              value: >-
                bundled,oneapi,jwt,oidc,jwtissuer,appauthoriser,apiauthoriser,pathfinder,fcjwtissuer,cookies-to-headers,opentelemetry
            - name: KONG_PLUGINSERVER_JS_QUERY_CMD
              value: >-
                /usr/bin/kong-js-pluginserver --plugins-directory
                /usr/local/kong/js-plugins --dump-all-plugins
            - name: KONG_PLUGINSERVER_JS_SOCKET
              value: /usr/local/kong/js_pluginserver.sock
            - name: KONG_PLUGINSERVER_JS_START_CMD
              value: >-
                /usr/bin/kong-js-pluginserver -v --plugins-directory
                /usr/local/kong/js-plugins
            - name: KONG_PLUGINSERVER_NAMES
              value: js
            - name: KONG_PORTAL_API_ACCESS_LOG
              value: /dev/stdout
            - name: KONG_PORTAL_API_ERROR_LOG
              value: /dev/stderr
            - name: KONG_PORT_MAPS
              value: '80:8000'
            - name: KONG_PREFIX
              value: /kong_prefix/
            - name: KONG_PROXY_ACCESS_LOG
              value: /dev/stdout
            - name: KONG_PROXY_ERROR_LOG
              value: /dev/stderr
            - name: KONG_PROXY_LISTEN
              value: 0.0.0.0:8000, [::]:8000
            - name: KONG_PROXY_STREAM_ACCESS_LOG
              value: /dev/stdout basic
            - name: KONG_PROXY_STREAM_ERROR_LOG
              value: /dev/stderr
            - name: KONG_ROUTER_FLAVOR
              value: traditional
            - name: KONG_STATUS_ACCESS_LOG
              value: 'off'
            - name: KONG_STATUS_ERROR_LOG
              value: /dev/stderr
            - name: KONG_STATUS_LISTEN
              value: 0.0.0.0:8100, [::]:8100
            - name: KONG_STREAM_LISTEN
              value: 'off'
            - name: KONG_TRACING_INSTRUMENTATIONS
              value: all
            - name: KONG_TRACING_SAMPLING_RATE
              value: '1.0'
            - name: KONG_UNTRUSTED_LUA_SANDBOX_REQUIRES
              value: resty.http,cjson,ngx.base64
          resources:
            limits:
              memory: 1019Mi
            requests:
              cpu: 17m
              memory: 1019Mi
          volumeMounts:
            - name: kong-prefix-dir
              mountPath: /kong_prefix/
            - name: kong-tmp
              mountPath: /tmp
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
        - name: wait-for-db
          image: kong:382
          args:
            - /bin/bash
            - '-c'
            - >-
              export KONG_NGINX_DAEMON=on KONG_PREFIX=`mktemp -d`
              KONG_KEYRING_ENABLED=off; until kong start; do echo 'waiting for
              db'; sleep 1; done; kong stop
          env:
            - name: KONG_ADMIN_ACCESS_LOG
              value: /dev/stdout
            - name: KONG_ADMIN_ERROR_LOG
              value: /dev/stderr
            - name: KONG_ADMIN_GUI_ACCESS_LOG
              value: /dev/stdout
            - name: KONG_ADMIN_GUI_ERROR_LOG
              value: /dev/stderr
            - name: KONG_ADMIN_LISTEN
              value: 0.0.0.0:8001, [::]:8001
            - name: KONG_CLUSTER_LISTEN
              value: 'off'
            - name: KONG_DATABASE
              value: postgres
            - name: KONG_LOG_LEVEL
              value: notice
            - name: KONG_NGINX_PROXY_LARGE_CLIENT_HEADER_BUFFERS
              value: 8 24k
            - name: KONG_LUA_PACKAGE_PATH
              value: /opt/?.lua;/opt/?/init.lua;;
            - name: KONG_NGINX_WORKER_PROCESSES
              value: '5'
            - name: KONG_ONEAPI_GATEWAY_KEY
              value: 
            - name: KONG_ONEAPI_GATEWAY_SECRET
              value: 
            - name: KONG_ONEAPI_HTTPS_PROXY
              value: http://:3128
            - name: KONG_PG_DATABASE
              value: kong
            - name: KONG_PG_HOST
              value: kong-db
            - name: KONG_PG_PASSWORD
              value: 
            - name: KONG_PG_PORT
              value: '64000'
            - name: KONG_PG_SCHEMA
              value: public
            - name: KONG_PG_SSL
              value: 'off'
            - name: KONG_PG_SSL_VERIFY
              value: 'off'
            - name: KONG_PG_USER
              value: kong
            - name: KONG_PLUGINS
              value: >-
                bundled,oneapi,jwt,jwtissuer,appauthoriser,apiauthoriser,pathfinder,fcjwtissuer,opentelemetry
            - name: KONG_PLUGINSERVER_JS_QUERY_CMD
              value: >-
                /usr/bin/kong-js-pluginserver --plugins-directory
                /usr/local/kong/js-plugins --dump-all-plugins
            - name: KONG_PLUGINSERVER_JS_SOCKET
              value: /usr/local/kong/js_pluginserver.sock
            - name: KONG_PLUGINSERVER_JS_START_CMD
              value: >-
                /usr/bin/kong-js-pluginserver -v --plugins-directory
                /usr/local/kong/js-plugins
            - name: KONG_PLUGINSERVER_NAMES
              value: js
            - name: KONG_PORTAL_API_ACCESS_LOG
              value: /dev/stdout
            - name: KONG_PORTAL_API_ERROR_LOG
              value: /dev/stderr
            - name: KONG_PORT_MAPS
              value: '80:8000'
            - name: KONG_PREFIX
              value: /kong_prefix/
            - name: KONG_PROXY_ACCESS_LOG
              value: /dev/stdout
            - name: KONG_PROXY_ERROR_LOG
              value: /dev/stderr
            - name: KONG_PROXY_LISTEN
              value: 0.0.0.0:8000, [::]:8000
            - name: KONG_PROXY_STREAM_ACCESS_LOG
              value: /dev/stdout basic
            - name: KONG_PROXY_STREAM_ERROR_LOG
              value: /dev/stderr
            - name: KONG_ROUTER_FLAVOR
              value: traditional
            - name: KONG_STATUS_ACCESS_LOG
              value: 'off'
            - name: KONG_STATUS_ERROR_LOG
              value: /dev/stderr
            - name: KONG_STATUS_LISTEN
              value: 0.0.0.0:8100, [::]:8100
            - name: KONG_STREAM_LISTEN
              value: 'off'
            - name: KONG_TRACING_INSTRUMENTATIONS
              value: all
            - name: KONG_TRACING_SAMPLING_RATE
              value: '1.0'
            - name: KONG_UNTRUSTED_LUA_SANDBOX_REQUIRES
              value: resty.http,cjson,ngx.base64
          resources:
            limits:
              memory: 1019Mi
            requests:
              cpu: 17m
              memory: 1019Mi
          volumeMounts:
            - name: kong-prefix-dir
              mountPath: /kong_prefix/
            - name: kong-tmp
              mountPath: /tmp
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      containers:
        - name: simple-rp
          image: kong-reverse-proxy:2
          ports:
            - containerPort: 8080
              protocol: TCP
          env:
            - name: HTTPS_PROXY
              value: http://:3128
            - name: HTTP_PROXY
              value: http://:3128
            - name: NO_PROXY
              value: >-
                
          resources:
            limits:
              memory: 100Mi
            requests:
              cpu: 10m
              memory: 100Mi
          livenessProbe:
            httpGet:
              path: /alive
              port: 8080
              scheme: HTTP
              httpHeaders:
                - name: x-alive
                  value: healthcheck
            initialDelaySeconds: 10
            timeoutSeconds: 1
            periodSeconds: 5
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
        - name: proxy
          image: kong:382
          ports:
            - name: admin
              containerPort: 8001
              protocol: TCP
            - name: proxy
              containerPort: 8000
              protocol: TCP
            - name: status
              containerPort: 8100
              protocol: TCP
          env:
            - name: KONG_ADMIN_ACCESS_LOG
              value: /dev/stdout
            - name: KONG_ADMIN_ERROR_LOG
              value: /dev/stderr
            - name: KONG_ADMIN_GUI_ACCESS_LOG
              value: /dev/stdout
            - name: KONG_ADMIN_GUI_ERROR_LOG
              value: /dev/stderr
            - name: KONG_ADMIN_LISTEN
              value: 0.0.0.0:8001, [::]:8001
            - name: KONG_CLUSTER_LISTEN
              value: 'off'
            - name: KONG_DATABASE
              value: postgres
            - name: KONG_LOG_LEVEL
              value: debug
            - name: KONG_LUA_PACKAGE_PATH
              value: /opt/?.lua;/opt/?/init.lua;;
            - name: KONG_NGINX_PROXY_LARGE_CLIENT_HEADER_BUFFERS
              value: 8 24k
            - name: KONG_NGINX_WORKER_PROCESSES
              value: '5'
            - name: KONG_ONEAPI_GATEWAY_KEY
              value: 
            - name: KONG_ONEAPI_GATEWAY_SECRET
              value: 
            - name: KONG_ONEAPI_HTTPS_PROXY
              value: http://:3128
            - name: KONG_PG_DATABASE
              value: kong
            - name: KONG_PG_HOST
              value: kong-db
            - name: KONG_PG_PASSWORD
              value: 
            - name: KONG_PG_PORT
              value: '64000'
            - name: KONG_PG_SCHEMA
              value: public
            - name: KONG_PG_SSL
              value: 'off'
            - name: KONG_PG_SSL_VERIFY
              value: 'off'
            - name: KONG_PG_USER
              value: kong
            - name: KONG_PLUGINS
              value: >-
                bundled,oneapi,jwt,oidc,jwtissuer,appauthoriser,apiauthoriser,pathfinder,fcjwtissuer,cookies-to-headers,opentelemetry
            - name: KONG_PLUGINSERVER_JS_QUERY_CMD
              value: >-
                /usr/bin/kong-js-pluginserver --plugins-directory
                /usr/local/kong/js-plugins --dump-all-plugins
            - name: KONG_PLUGINSERVER_JS_SOCKET
              value: /usr/local/kong/js_pluginserver.sock
            - name: KONG_PLUGINSERVER_JS_START_CMD
              value: >-
                /usr/bin/kong-js-pluginserver -v --plugins-directory
                /usr/local/kong/js-plugins
            - name: KONG_PLUGINSERVER_NAMES
              value: js
            - name: KONG_PORTAL_API_ACCESS_LOG
              value: /dev/stdout
            - name: KONG_PORTAL_API_ERROR_LOG
              value: /dev/stderr
            - name: KONG_PORT_MAPS
              value: '80:8000'
            - name: KONG_PREFIX
              value: /kong_prefix/
            - name: KONG_PROXY_ACCESS_LOG
              value: /dev/stdout
            - name: KONG_PROXY_ERROR_LOG
              value: /dev/stderr
            - name: KONG_PROXY_LISTEN
              value: 0.0.0.0:8000, [::]:8000
            - name: KONG_PROXY_STREAM_ACCESS_LOG
              value: /dev/stdout basic
            - name: KONG_PROXY_STREAM_ERROR_LOG
              value: /dev/stderr
            - name: KONG_ROUTER_FLAVOR
              value: traditional
            - name: KONG_STATUS_ACCESS_LOG
              value: 'off'
            - name: KONG_STATUS_ERROR_LOG
              value: /dev/stderr
            - name: KONG_STATUS_LISTEN
              value: 0.0.0.0:8100, [::]:8100
            - name: KONG_STREAM_LISTEN
              value: 'off'
            - name: KONG_TRACING_INSTRUMENTATIONS
              value: all
            - name: KONG_TRACING_SAMPLING_RATE
              value: '1.0'
            - name: KONG_UNTRUSTED_LUA_SANDBOX_REQUIRES
              value: resty.http,cjson,ngx.base64
            - name: KONG_NGINX_DAEMON
              value: 'off'
          resources:
            limits:
              memory: 1019Mi
            requests:
              cpu: 17m
              memory: 1019Mi
          volumeMounts:
            - name: kong-prefix-dir
              mountPath: /kong_prefix/
            - name: kong-tmp
              mountPath: /tmp
          livenessProbe:
            httpGet:
              path: /status
              port: status
              scheme: HTTP
            initialDelaySeconds: 5
            timeoutSeconds: 5
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /status/ready
              port: status
              scheme: HTTP
            initialDelaySeconds: 5
            timeoutSeconds: 5
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          lifecycle:
            preStop:
              exec:
                command:
                  - kong
                  - quit
                  - '--wait=15'
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: kong
      serviceAccount: kong
      automountServiceAccountToken: false
      securityContext:
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001
        seccompProfile:
          type: RuntimeDefault
      imagePullSecrets:
        - name: imagepullsecret
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600