Config errors in ingress controller log 1.3 rc01 db-less

Seeing the following errors in the ingress controller logs. When these errors occur any load testing going on generates a slew of 404 responses to the client. We have multiple ingress controllers in the namespace (kubernetes deployment). The ingress controller is configured to watch-namespace and has and ingress.class custom setting. many of these configurations are from entities that do not have the ingress.class set to our custom setting.
Is there some basic configuration we are missing here?

Containers:
ingress-controller:
Container ID: **************************************************************************
Image: kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller:0.5.0
Image ID: docker-pullable://kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller@sha256:d0b53e95c9772d41a8031d7ade6b3958822fe3e8747a31ff6c2f686b1c3df2a9
Port:
Host Port:
Args:
/kong-ingress-controller
–publish-service=sandbox/kong-sbox-kong-proxy
–ingress-class=kong-sbox
–election-id=kong-ingress-controller-leader-kong-sbox
–watch-namespace=sandbox
–kong-url=https://localhost:8444
–admin-tls-skip-verify
State: Running
Started: Tue, 06 Aug 2019 18:11:17 -0400
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Tue, 06 Aug 2019 18:11:15 -0400
Finished: Tue, 06 Aug 2019 18:11:15 -0400
Ready: True
Restart Count: 1
Limits:
cpu: 4
memory: 8Gi
Requests:
cpu: 1
memory: 4Gi
Liveness: http-get http://:10254/healthz delay=30s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: kong-sbox-kong-664945d9c-9d6xh (v1:metadata.name)
POD_NAMESPACE: sandbox (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kong-sbox-kong-token-j7pk9 (ro)

I0807 20:27:21.073584 1 controller.go:128] successfully synced configuration to Kong
I0807 20:29:18.534463 1 controller.go:128] successfully synced configuration to Kong
I0807 20:30:46.747236 1 controller.go:128] successfully synced configuration to Kong
I0807 20:31:18.103947 1 controller.go:128] successfully synced configuration to Kong
E0807 20:31:21.366368 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:21.366387 1 queue.go:113] requeuing sandbox/echo-service-echo-sandbox, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:24.693851 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:24.693870 1 queue.go:113] requeuing sandbox/echo-service-via-sandbox, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:28.024281 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:28.024296 1 queue.go:113] requeuing sandbox/via-test-api, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:31.358739 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:31.358757 1 queue.go:113] requeuing sandbox/aspect-via-no-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:34.694545 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:34.694562 1 queue.go:113] requeuing sandbox/echo-unit-test-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:38.067499 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:38.067515 1 queue.go:113] requeuing sandbox/aspect-auth, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:41.359801 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:41.359816 1 queue.go:113] requeuing sandbox/aspect-via-x-api-key-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:44.692229 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:44.692250 1 queue.go:113] requeuing sandbox/echo-load-test-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:48.024557 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:48.024574 1 queue.go:113] requeuing sandbox/via-test-api-rewrite-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:51.357787 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:51.357804 1 queue.go:113] requeuing sandbox/via-test-api-external-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:54.690705 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:54.690720 1 queue.go:113] requeuing sandbox/echo-rewrite-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:58.031202 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:58.031220 1 queue.go:113] requeuing sandbox/aspect-via-development-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:01.401982 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:01.402000 1 queue.go:113] requeuing sandbox/aspect-datadog, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:04.692381 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:04.692397 1 queue.go:113] requeuing sandbox/echo-load-test-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:08.024633 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:08.024650 1 queue.go:113] requeuing sandbox/aspect-via-development-rate-limit-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:11.357638 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:11.357654 1 queue.go:113] requeuing sandbox/echo-unit-test-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:14.691826 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:14.691844 1 queue.go:113] requeuing sandbox/aspect-via-external-rate-limit-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:18.024860 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:18.024874 1 queue.go:113] requeuing sandbox/aspect-via-no-rate-limit-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:21.392017 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:21.392033 1 queue.go:113] requeuing sandbox/via-test-api, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:24.709231 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:24.709246 1 queue.go:113] requeuing sandbox/configuration-services, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:28.026376 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:28.026393 1 queue.go:113] requeuing sandbox/echo-service-echo-sandbox, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:31.357799 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:31.357815 1 queue.go:113] requeuing sandbox/echo-service-via-sandbox, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:34.692428 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:34.692445 1 queue.go:113] requeuing sandbox/via-test-api, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:38.024193 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:38.024208 1 queue.go:113] requeuing sandbox/aspect-via-no-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:41.362182 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:41.362199 1 queue.go:113] requeuing sandbox/echo-unit-test-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:44.738256 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:44.738273 1 queue.go:113] requeuing sandbox/aspect-auth, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:48.026913 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:48.026929 1 queue.go:113] requeuing sandbox/aspect-via-x-api-key-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:51.358638 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:51.358656 1 queue.go:113] requeuing sandbox/echo-load-test-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:54.691906 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:54.691923 1 queue.go:113] requeuing sandbox/via-test-api-rewrite-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:58.024295 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:58.024309 1 queue.go:113] requeuing sandbox/via-test-api-external-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:33:01.357381 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:33:01.357400 1 queue.go:113] requeuing sandbox/echo-rewrite-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:33:04.697184 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:33:04.697200 1 queue.go:113] requeuing sandbox/aspect-via-development-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:33:08.069090 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:33:08.069116 1 queue.go:113] requeuing sandbox/aspect-datadog, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:33:11.360041 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:33:11.360056 1 queue.go:113] requeuing sandbox/echo-load-test-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:33:14.691888 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:33:14.691907 1 queue.go:113] requeuing sandbox/aspect-via-development-rate-limit-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:33:18.024680 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:33:18.024698 1 queue.go:113] requeuing sandbox/echo-unit-test-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
I0807 20:33:21.395365 1 controller.go:128] successfully synced configuration to Kong
I0807 20:33:24.720876 1 controller.go:128] successfully synced configuration to Kong
I0807 20:33:28.075981 1 controller.go:128] successfully synced configuration to Kong
I0807 20:34:56.692747 1 controller.go:128] successfully synced configuration to Kong
I0807 20:34:56.780817 1 controller.go:128] successfully synced configuration to Kong
I0807 20:38:18.487458 1 controller.go:128] successfully synced configuration to Kong
I0807 20:40:34.387398 1 controller.go:128] successfully synced configuration to Kong

Hi @bobs! Could you share the output of Kong’s error.log? These 500 Internal Server Error {“message”:“An unexpected error occurred”} should have generated tracebacks for the error on the Kong side. That would help us immensely to diagnose this! Thanks!

Sure thing. We have switched back to db because of the issue. I will see if we can switch back to db-less and grab that info for you. May be a couple of days before I have the opportunity.


© 2018 Kong Inc.    Terms  •  Privacy  •  FAQ