we just set up Kong cluster with 2 kong nodes on our production environment. We didn’t use the load balancer in front of Kong , but using Nginx service on one of Kong nodes to proxy requests to Kong nodes
While we did the pressure test using Apach ab tool ,we found a lot of failed requests .
[work@DWD-BETA ~]$ab -n 1000 -c 100 "http://www,abc.com/api/" This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking www.abc.com (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Completed 500 requests Completed 600 requests Completed 700 requests Completed 800 requests Completed 900 requests Completed 1000 requests Finished 1000 requests Server Software: Server Hostname: www.abc.com Server Port: 80 Document Path: /api Document Length: 206 bytes Concurrency Level: 100 Time taken for tests: 7.894 seconds Complete requests: 1000 Failed requests: 599 (Connect: 0, Receive: 0, Length: 599, Exceptions: 0) Write errors: 0 Non-2xx responses: 401 Total transferred: 599127 bytes HTML transferred: 365933 bytes Requests per second: 126.67 [#/sec] (mean) Time per request: 789.428 [ms] (mean) Time per request: 7.894 [ms] (mean, across all concurrent requests) Transfer rate: 74.12 [Kbytes/sec] received
after checking the kong proxy access logs ,I found a lot of 503 errors:
xx.xx.xx.xx - - [21/Dec/2018:16:24:47 +0800] "GET /api/ HTTP/1.0" 503 206 "-" "-" "ApacheBench/2.3 "xx.xx.xx.xx"-" 0.001 "0.001"
I did some optimizations for kong Nginx . the nginx.conf files as below:
But it didn’t work .
Then I try to using a load balancer to proxy requests to Kong nodes.But the problem still happens.
This problem disappeared if the load balancer proxy requests directly to the real servers on the backend.
The Kong version is the latest(v1.14) and the cassandra database is 3.11.3
I have been troubleshooting this problem for a whole day , and there is no any helpful information on Google.
Could anyone help me please ?
Thank you very much in advance. Forgive my bad English!