So been giving building from SRC and hacking in the openresty bit into the existing Kong tarball. We tried the below:
#From Source with Custom PATCH Approach
RUN apk add --no-cache --virtual build-deps wget tar ca-certificates \
&& apk add --update --no-cache readline-dev linux-headers build-deps make gcc g++ libgcc zlib-dev perl pcre pcre-dev unzip tzdata luarocks git
RUN wget https://openresty.org/download/openresty-1.13.6.2.tar.gz \
&& tar -zxvf openresty-1.13.6.2.tar.gz -C /tmp \
&& wget https://github.com/Kong/openresty-patches/archive/master.tar.gz \
&& tar -zxvf master.tar.gz -C /tmp \
&& cd /tmp/openresty-1.13.6.2/bundle \
&& wget https://github.company.com/raw/MyFunOrg/optum-kong-docker/dev/patches/ngx/nginx-1.13.6_07-upstream-keepalive.patch -P /tmp/openresty-patches-master/patches/1.13.6.2/ \
&& cd /tmp/openresty-1.13.6.2/bundle && for i in /tmp/openresty-patches-master/patches/1.13.6.2/*.patch; do patch -p1 < $i; done
RUN cd /tmp \
&& curl -fSL https://www.openssl.org/source/openssl-1.1.1b.tar.gz -o openssl-1.1.1b.tar.gz \
&& tar xzf openssl-1.1.1b.tar.gz
RUN cd /tmp/openresty-1.13.6.2 \
&& ./configure -j2 --with-openssl=/tmp/openssl-1.1.1b --with-pcre-jit --with-http_ssl_module --with-http_realip_module --with-http_stub_status_module --with-http_v2_module \
&& make -j2 \
&& make install
#Add valgrind for mem leak checks, iproute2 for ss util, apache2-utils to second apk add line 9 if you ever want to test ab(apache benchmark) for localhost testing to bypass routers.
RUN apk add --no-cache --virtual .build-deps wget tar ca-certificates \
&& apk add --no-cache libgcc openssl pcre perl unzip tzdata luarocks git \
&& wget -O kong.tar.gz "https://bintray.com/kong/kong-alpine-tar/download_file?file_path=kong-$KONG_VERSION.apk.tar.gz" \
&& echo "$KONG_SHA256 *kong.tar.gz" | sha256sum -c - \
&& tar -xzf kong.tar.gz -C /tmp \
&& rm -f kong.tar.gz \
&& rm -rf /tmp/usr/local/openresty \
&& cp -R /tmp/usr / \
&& rm -rf /tmp/usr \
&& cp -R /tmp/etc / \
&& rm -rf /tmp/etc \
&& apk del .build-deps
We currently get this error on the deploy:
ERROR: error loading module '_openssl.bignum' from file '/usr/local/lib/lua/5.1/_openssl.so':
Error loading shared library libssl.so.1.1: No such file or directory (needed by /usr/local/lib/lua/5.1/_openssl.so)
stack traceback:
[C]: in function 'require'
/usr/local/share/lua/5.1/openssl/bignum.lua:1: in main chunk
[C]: in function 'require'
/usr/local/share/lua/5.1/kong/cmd/utils/prefix_handler.lua:4: in main chunk
[C]: in function 'require'
/usr/local/share/lua/5.1/kong/cmd/prepare.lua:1: in main chunk
[C]: in function 'require'
/usr/local/share/lua/5.1/kong/cmd/init.lua:54: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:44>
/usr/local/bin/kong:7: in function 'file_gen'
init_worker_by_lua:50: in function <init_worker_by_lua:48>
[C]: in function 'xpcall'
init_worker_by_lua:57: in function <init_worker_by_lua:55>
Does anybody have any ideas on how to fix this, is this a library linking problem? Note the
&& rm -rf /tmp/usr/local/openresty \
which deviates from the normal docker-kong approach.
Testing this on Kong 1.2.0rc2 right now.
We then thought maybe adding an openssl from source pointing the config to /usr/local/kong like so might help prior to the openresty compile, but we still got the same error at deploy as in my original post.
RUN cd /tmp \
&& curl -fSL https://www.openssl.org/source/openssl-1.1.1b.tar.gz -o openssl-1.1.1b.tar.gz \
&& tar xzf openssl-1.1.1b.tar.gz \
&& cd openssl-1.1.1b \
&& eval ./config --prefix=/usr/local/kong --openssldir=/usr/local/kong -fPIC shared \
&& make -j2 \
&& make install_sw
@jeremyjpj0916 In the first approach you used (which has less margin for errors), the issue is with luaossl (a Lua OpenSSL binding we use) not being able to find the OpenSSL shared lib. This should be handled by LuaRocks. When you install Kong’s LuaRocks dependencies, make sure that you specify the proper path for OPENSSL_DIR and OPENSSL_LIBDIR environment variables, like so:
@thibaultcha
We managed to make it work with the below code, its probably messy and not perfect but it did deploy and start running:
#From Source with Custom PATCH Approach
RUN apk add --no-cache --virtual build-deps wget tar ca-certificates \
&& apk add --update --no-cache readline-dev outils-md5 linux-headers bsd-compat-headers m4 yaml-dev build-deps make gcc g++ libgcc zlib-dev perl pcre pcre-dev unzip tzdata luarocks git
#Pull Openresty and patch
RUN wget https://openresty.org/download/openresty-1.13.6.2.tar.gz \
&& tar -zxvf openresty-1.13.6.2.tar.gz -C /tmp \
&& wget https://github.com/Kong/openresty-patches/archive/master.tar.gz \
&& tar -zxvf master.tar.gz -C /tmp \
&& cd /tmp/openresty-1.13.6.2/bundle \
&& wget https://github.company.com/raw/FunRepo/optum-kong-docker/dev/patches/ngx/nginx-1.13.6_07-upstream-keepalive.patch -P /tmp/openresty-patches-master/patches/1.13.6.2/ \
&& cd /tmp/openresty-1.13.6.2/bundle && for i in /tmp/openresty-patches-master/patches/1.13.6.2/*.patch; do patch -p1 < $i; done
#Dowload OpenSSL
RUN cd /tmp \
&& curl -fSL https://www.openssl.org/source/openssl-1.1.1b.tar.gz -o openssl-1.1.1b.tar.gz \
&& tar xzf openssl-1.1.1b.tar.gz
#Build OpenResty from source
RUN cd /tmp/openresty-1.13.6.2 \
&& ./configure -j2 --with-openssl=/tmp/openssl-1.1.1b --with-pcre-jit --with-http_ssl_module --with-http_realip_module --with-http_stub_status_module --with-http_v2_module \
&& make -j2 \
&& make install
#Build Luarocks from source
RUN cd /tmp \
&& wget https://luarocks.org/releases/luarocks-3.1.3.tar.gz \
&& tar zxpf luarocks-3.1.3.tar.gz \
&& cd luarocks-3.1.3 \
&& ./configure --lua-suffix=jit --with-lua=/usr/local/openresty/luajit --with-lua-include=/usr/local/openresty/luajit/include/luajit-2.1 \
&& make build \
&& make install
#Needed for luarocks dependency builds to force https vs ssh
RUN git config --global url.https://github.com/.insteadOf git://github.com/
#Install Kong via luarocks
RUN luarocks install kong 1.1.2-0 OPENSSL_DIR=/tmp/openssl-1.1.1b OPENSSL_LIBDIR=/tmp/openssl-1.1.1b CRYPTO_DIR=/tmp/openssl-1.1.1b/.openssl
#Get the Kong executable resty script
RUN wget https://raw.githubusercontent.com/Kong/kong/master/bin/kong -P /usr/bin
RUN chmod 777 /usr/bin/kong
RUN mv /usr/local/openresty/bin/resty /usr/bin
Now for the details:
-
Yes Kong with this patch is managing upstream keepalive and taking the hostname into account for the connection pool so doing proper keepalive tested by running a backend with passthrough and backend with edge differing subdomains on same ip:port pair proxying through kong at same time. The performance is seemingly great from what we can tell, ephemeral port exhaustion at high throughput is no longer occurring on our Kong node. - Can’t emphasize how big a deal this is, it lets us run kong as desired with keepalive to backends.
-
Only hitch we currently see in behavior is with our frontend LB we now see some connection RST making their way back to the originating client we used to never see before. We do not observe this if we call an exposed passthrough route on our cloud cluster(removing the LB from the equation). This did not occur on Kong before applying this patch it had seemed. Seems TPS does not matter either. Its very intermittent and very reproducible through the lb, anywhere from 0 to 100%. We also have seen our LB health checks fail sporadically from the lb too which is causing some of it, the question is why are the healthchecks failing now, some kinda shared resource around sockets… not sure. To picture it out:
RST issues:
LB -> Cloud Ingress -> Kong Pod -> Backend (RST issues to client and LB fails heathchecks sometimes)
NO RST ISSUES:
Exposed route on Cloud Ingress directly -> Kong Pod -> Backend
Even with the current RST issue, I feel we can probably overcome that and this is probably the best news around integrations with Kong we could have all year here. Thanks again for writing up that patch!
Maybe one day this can be an optional config choice in kong and added to the openresty-patches repo eh 
? We could make a PR to your repo to get that ball rolling sometime if Kong wants to enhance its upstream keepalive behavior for all Kong users. This patch fixes what in nerd speak has been the kryptonite to us of Kong as an API Gateway!
@jeremyjpj0916 Have you had any luck tracking down the client RST packets so far?
@thibaultcha
Updates around this, we were testing in a dev lab. This lab runs in 2 data centers with a LB in each. We found the RST’s to only occur from one DC and works flawlessly calling through the other DC. We are testing in a secondary Stage env now both datacenters and their respective LB’s. So far testing there is not producing the RST’s .
If we confirm no issue in Stage we will roll it out to Stage in all network zones for 2-3 weeks. If 2-3 weeks yields no issue, we would then roll it out to prod. Only downside is our docker builds now make us wait like 13 minutes building from src vs the nice kong tarball package that takes like 1-2 minutes to setup 
. Will keep you in the loop!
We will do some followup around the one dev lab in a specific dc we see the RST but I think it will be safe to say if all other stage environments face no issue it has to be something wrong with our LB configs there and not related to the patch(as patch should only impact upstream connections anyway, not client connections).
1 Like
@thibaultcha We have indeed isolated it to a bad piece of LB infrastructure in our dev environment, the time the LB it actually started impacting us was the 4th of June so long before applying the custom patch change, so the patch change has no relevancy to the RST issue(which we are still looking into but rest assured its not related, seems the RSTs occur every 5 mins on the dot almost which is odd). But yeah we now have the patch running in all our dev and stage environments and so far so good . Looking to roll out to production likely 7/1 or so, very promising results thus far in terms of continued good performance and the issue of improper upstream routing alleviated.
EDIT - Slight issue discovered with the upstream patch logic: https://github.com/Kong/openresty-patches/pull/40#issuecomment-503339958