Seeing the following errors in the ingress controller logs. When these errors occur any load testing going on generates a slew of 404 responses to the client. We have multiple ingress controllers in the namespace (kubernetes deployment). The ingress controller is configured to watch-namespace and has and ingress.class custom setting. many of these configurations are from entities that do not have the ingress.class set to our custom setting.
Is there some basic configuration we are missing here?
Containers:
ingress-controller:
Container ID: **************************************************************************
Image: kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller:0.5.0
Image ID: docker-pullable://kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller@sha256:d0b53e95c9772d41a8031d7ade6b3958822fe3e8747a31ff6c2f686b1c3df2a9
Port:
Host Port:
Args:
/kong-ingress-controller
–publish-service=sandbox/kong-sbox-kong-proxy
–ingress-class=kong-sbox
–election-id=kong-ingress-controller-leader-kong-sbox
–watch-namespace=sandbox
–kong-url=https://localhost:8444
–admin-tls-skip-verify
State: Running
Started: Tue, 06 Aug 2019 18:11:17 -0400
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Tue, 06 Aug 2019 18:11:15 -0400
Finished: Tue, 06 Aug 2019 18:11:15 -0400
Ready: True
Restart Count: 1
Limits:
cpu: 4
memory: 8Gi
Requests:
cpu: 1
memory: 4Gi
Liveness: http-get http://:10254/healthz delay=30s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: kong-sbox-kong-664945d9c-9d6xh (v1:metadata.name)
POD_NAMESPACE: sandbox (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kong-sbox-kong-token-j7pk9 (ro)
I0807 20:27:21.073584 1 controller.go:128] successfully synced configuration to Kong
I0807 20:29:18.534463 1 controller.go:128] successfully synced configuration to Kong
I0807 20:30:46.747236 1 controller.go:128] successfully synced configuration to Kong
I0807 20:31:18.103947 1 controller.go:128] successfully synced configuration to Kong
E0807 20:31:21.366368 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:21.366387 1 queue.go:113] requeuing sandbox/echo-service-echo-sandbox, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:24.693851 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:24.693870 1 queue.go:113] requeuing sandbox/echo-service-via-sandbox, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:28.024281 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:28.024296 1 queue.go:113] requeuing sandbox/via-test-api, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:31.358739 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:31.358757 1 queue.go:113] requeuing sandbox/aspect-via-no-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:34.694545 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:34.694562 1 queue.go:113] requeuing sandbox/echo-unit-test-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:38.067499 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:38.067515 1 queue.go:113] requeuing sandbox/aspect-auth, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:41.359801 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:41.359816 1 queue.go:113] requeuing sandbox/aspect-via-x-api-key-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:44.692229 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:44.692250 1 queue.go:113] requeuing sandbox/echo-load-test-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:48.024557 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:48.024574 1 queue.go:113] requeuing sandbox/via-test-api-rewrite-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:51.357787 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:51.357804 1 queue.go:113] requeuing sandbox/via-test-api-external-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:54.690705 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:54.690720 1 queue.go:113] requeuing sandbox/echo-rewrite-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:31:58.031202 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:31:58.031220 1 queue.go:113] requeuing sandbox/aspect-via-development-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:01.401982 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:01.402000 1 queue.go:113] requeuing sandbox/aspect-datadog, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:04.692381 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:04.692397 1 queue.go:113] requeuing sandbox/echo-load-test-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:08.024633 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:08.024650 1 queue.go:113] requeuing sandbox/aspect-via-development-rate-limit-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:11.357638 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:11.357654 1 queue.go:113] requeuing sandbox/echo-unit-test-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:14.691826 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:14.691844 1 queue.go:113] requeuing sandbox/aspect-via-external-rate-limit-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:18.024860 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:18.024874 1 queue.go:113] requeuing sandbox/aspect-via-no-rate-limit-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:21.392017 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:21.392033 1 queue.go:113] requeuing sandbox/via-test-api, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:24.709231 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:24.709246 1 queue.go:113] requeuing sandbox/configuration-services, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:28.026376 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:28.026393 1 queue.go:113] requeuing sandbox/echo-service-echo-sandbox, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:31.357799 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:31.357815 1 queue.go:113] requeuing sandbox/echo-service-via-sandbox, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:34.692428 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:34.692445 1 queue.go:113] requeuing sandbox/via-test-api, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:38.024193 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:38.024208 1 queue.go:113] requeuing sandbox/aspect-via-no-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:41.362182 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:41.362199 1 queue.go:113] requeuing sandbox/echo-unit-test-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:44.738256 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:44.738273 1 queue.go:113] requeuing sandbox/aspect-auth, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:48.026913 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:48.026929 1 queue.go:113] requeuing sandbox/aspect-via-x-api-key-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:51.358638 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:51.358656 1 queue.go:113] requeuing sandbox/echo-load-test-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:54.691906 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:54.691923 1 queue.go:113] requeuing sandbox/via-test-api-rewrite-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:32:58.024295 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:32:58.024309 1 queue.go:113] requeuing sandbox/via-test-api-external-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:33:01.357381 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:33:01.357400 1 queue.go:113] requeuing sandbox/echo-rewrite-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:33:04.697184 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:33:04.697200 1 queue.go:113] requeuing sandbox/aspect-via-development-rate-limit-plugin, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:33:08.069090 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:33:08.069116 1 queue.go:113] requeuing sandbox/aspect-datadog, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:33:11.360041 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:33:11.360056 1 queue.go:113] requeuing sandbox/echo-load-test-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:33:14.691888 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:33:14.691907 1 queue.go:113] requeuing sandbox/aspect-via-development-rate-limit-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
E0807 20:33:18.024680 1 controller.go:125] unexpected failure updating Kong configuration:
posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
W0807 20:33:18.024698 1 queue.go:113] requeuing sandbox/echo-unit-test-consumer, err posting new config to /config: 500 Internal Server Error {“message”:“An unexpected error occurred”}
I0807 20:33:21.395365 1 controller.go:128] successfully synced configuration to Kong
I0807 20:33:24.720876 1 controller.go:128] successfully synced configuration to Kong
I0807 20:33:28.075981 1 controller.go:128] successfully synced configuration to Kong
I0807 20:34:56.692747 1 controller.go:128] successfully synced configuration to Kong
I0807 20:34:56.780817 1 controller.go:128] successfully synced configuration to Kong
I0807 20:38:18.487458 1 controller.go:128] successfully synced configuration to Kong
I0807 20:40:34.387398 1 controller.go:128] successfully synced configuration to Kong