Hi
As far I know go plugins are binary files (.so) and they are bigger than ConfigMap/Secret size limit. So have can I load go plugin to kong in dbless mode?
For now, I’ve added sidecar that is coping go plugin file to kong container. But I’ve a problem with enabling it.
This is my vaules.yaml for kong helm chart.
` plugins:
configMaps:
- pluginName: transformer-http-go-plugin
name: <what should be here?>
This bums me out… I don’t really want to build a custom container just to add a single file (mostly because i don’t want to spin up the accompanying ci/cd and then host that container in some registry). Is there anything wrong with loading the golang library file similar to how we mount the lua files via configmaps into the container???
One needs to compile go and then put that inside the container.
This feature is like pealing an onion so we decided to not add in. I do understand that there is some other overhead.
I’d be open to solving problem with this solution as well but I don’t know of anyway to actually solve it.
Yeah, i think the issue of also needing to compile the go pluginserver adds an additional layer of complexity which complicates the “just using configmaps” method that the lua plugins use. I will continue to tinker around, but I suspect you are right, and the best way is just to make a custom container.
my attempt at creating a docker-file and adding my plugin:
FROM golang:1.14.0 as go-builder
# Build golang plugin(s)
WORKDIR /build
COPY ./custom-go-plugins ./
WORKDIR /build/go-hello/src
RUN go get -insecure ./...
RUN go build -buildmode=plugin .
# Build golang pdk
RUN go get github.com/Kong/go-pluginserver
FROM kong:2.0
WORKDIR /custom-go-plugins
# Add golang app
COPY --from=go-builder --chown=kong:root /build/go-hello/src/go-hello.so /custom-go-plugins/
# Add the pluginserver
ENV goPath=/go
COPY --from=go-builder --chown=kong:root ${goPath}/bin/go-pluginserver /usr/local/bin/
However, when i try to run the go-pluginserver inside the container i get a “no such file or directory error” I was suspicious that this error was actually the RPC socket file “/usr/local/kong/go_pluginserver.sock” but even if i manually make the file, i get the same error… So now i am suspicious I built this the wrong way, or I am missing some golang dependency… even though everything should be statically compiled since its go.
docker run -it bclouser/kong:2.0 /bin/bash
bash-5.0$ ls -lh /usr/local/bin/
total 14M
-rwxr-xr-x 1 kong root 13.9M May 13 04:39 go-pluginserver
-rwxr-xr-x 1 1000 1000 451 Apr 24 16:27 json2lua
-rwxr-xr-x 1 kong root 238 Apr 24 16:27 kong
-rwxr-xr-x 1 1000 1000 450 Apr 24 16:27 lapis
-rwxr-xr-x 1 1000 1000 451 Apr 24 16:27 lua2json
-rwxr-xr-x 1 1000 1000 1.4K Apr 24 16:26 luarocks
-rwxr-xr-x 1 1000 1000 964 Apr 24 16:26 luarocks-admin
bash-5.0$
bash-5.0$ /usr/local/bin/go-pluginserver
bash: /usr/local/bin/go-pluginserver: No such file or directory
The error in kong which led me to try running the pluginserver manually looks like this:
waiting for db
Error: /usr/local/share/lua/5.1/kong/cmd/start.lua:64: sh: /usr/local/bin/go-pluginserver: not found
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/MessagePack.lua:813: missing bytes
Note that your go-builder stage uses a Debian base image, while kong:2.0 is based on Alpine – as such, the binary you generate isn’t valid for the target system and you get that error message. Try using golang:1.14.0-alpine as Go builder and please let us know how it goes!
Well spotted @salazar! That fixed my issue with the pluginserver.
I do think i hit another issue with using both lua plugins and golang plugins side-by-side
waiting for db
Error: /usr/local/share/lua/5.1/kong/cmd/start.lua:64: 2020/05/13 23:31:06 failed to open plugin go-hello: plugin: not implemented
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:455: bad argument #1 to 'ipairs' (table expected, got nil)
I did have to add a hack to the ‘helpers.tpl’ file in order to get the ‘go-hello’ plugin added to the list of plugins, so I will continue tinkering around to see if this error is my fault, but the kong manifest output produced from a dry-run looks pretty good to me - the KONG_PLUGINS env block for the init-container and the proxy config looks like this:
Ah, ok, i figured it out. I was building the plugins without CGO, since in order to have cgo i had to apk add build-base and I assumed I didn’t need it… A few people belly-aching about it here: https://github.com/docker-library/golang/issues/153
Since go’s plugin package requires C functions, without CGO my plugin was probably causing a seg-fault or some kind of error inside the plugin.open() function which is why i saw failed to open plugin go-hello: plugin: not implemented
Things are working as expected now. Here is the final docker-file for building this:
FROM golang:1.14.2-alpine as go-builder
# Because the plugin package needs cgo (https://golang.org/pkg/plugin/)
# You can see some C code here: https://golang.org/src/plugin/plugin_dlopen.go
RUN apk add --no-cache build-base
# Build golang plugin(s)
WORKDIR /build
COPY ./custom-go-plugins ./
WORKDIR /build/go-hello/src
RUN go get -insecure ./...
RUN go build -buildmode=plugin .
# Build golang pluginserver
RUN go get github.com/Kong/go-pluginserver@v0.3.1
FROM kong:2.0
# Add golang app
COPY --from=go-builder --chown=kong:root /build/go-hello/src/go-hello.so /custom-go-plugins/
# Add the pluginserver
ENV goPath=/go
COPY --from=go-builder --chown=kong:root ${goPath}/bin/go-pluginserver /usr/local/bin/
And here is the patch I made to _helpers.tpl to allow adding golang plugins alongside my config-mapped LUA plugins:
--- a/kong/kong-1.5.0/templates/_helpers.tpl
+++ b/kong/kong-1.5.0/templates/_helpers.tpl
@@ -296,6 +296,9 @@ The name of the service used for the ingress controller's validation webhook
{{- define "kong.plugins" -}}
{{ $myList := list "bundled" }}
+{{- range .Values.plugins.goPlugins -}}
+{{- $myList = append $myList .pluginName -}}
+{{- end -}}
{{- range .Values.plugins.configMaps -}}
{{- $myList = append $myList .pluginName -}}
{{- end -}}
So my values.yaml file has a plugins block like this:
I am trying to follow your docker file only. Trying with go-hello plugin.
Able to start the kong successfully, but while executing the plugin, getting error. Follwing are the logs -
kong_1 | 2020/08/26 18:12:33 [crit] 26#0: *368 connect() to unix:/usr/local/kong/go_pluginserver.sock failed (2: No such file or directory), client: 172.19.0.1, server: kong, request: “GET /route1 HTTP/1.1”, host: “localhost:8000”
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *368 [kong] go.lua:140 [go-hello] trying to connect: no such file or directory, client: 172.19.0.1, server: kong, request: “GET /route1 HTTP/1.1”, host: “localhost:8000”
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *368 [kong] go.lua:429 [go-hello] starting instance: no such file or directory, client: 172.19.0.1, server: kong, request: “GET /route1 HTTP/1.1”, host: “localhost:8000”
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *368 lua coroutine: runtime error: /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:432: no such file or directory
kong_1 | stack traceback:
kong_1 | coroutine 0:
kong_1 | [C]: in function ‘error’
kong_1 | /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:432: in function ‘get_instance’
kong_1 | /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:498: in function </usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:497>
kong_1 | coroutine 1:
kong_1 | [C]: in function ‘resume’
kong_1 | coroutine.wrap:21: in function coroutine.wrap:21
kong_1 | /usr/local/share/lua/5.1/kong/init.lua:839: in function ‘access’
kong_1 | access_by_lua(nginx-kong.conf:91):2: in main chunk, client: 172.19.0.1, server: kong, request: “GET /route1 HTTP/1.1”, host: “localhost:8000”
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *368 [kong] init.lua:841 [go-hello] /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:432: no such file or directory, client: 172.19.0.1, server: kong, request: “GET /route1 HTTP/1.1”, host: “localhost:8000”
kong_1 | {“timestamp” : “26/Aug/2020:18:12:33 +0000”,“client_ip” : “172.19.0.1”,“real_ip”: “”,“host”: “localhost:8000”,“request” : {“method”: “GET”,“uri” : “/route1”,“request_length”: “647”,“x-siemens-correlation-id” : “”},“upstream” : “”,“response”: {“headers” : {“date” : “”,“X-Siemens-meter-data” : “”},“status” : 500,“size” : 42},“latencies” : {“request” : 0.011,“proxy” : “”},“rate limit” : 0,“connection_requests” : 1}
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *371 send() failed (111: Connection refused), context: ngx.timer, client: 172.19.0.1, server: 0.0.0.0:8000
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *371 [kong] statsd_logger.lua:76 failed to send data to localhost:8125: connection refused, context: ngx.timer, client: 172.19.0.1, server: 0.0.0.0:8000
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *371 send() failed (111: Connection refused), context: ngx.timer, client: 172.19.0.1, server: 0.0.0.0:8000
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *371 [kong] statsd_logger.lua:76 failed to send data to localhost:8125: connection refused, context: ngx.timer, client: 172.19.0.1, server: 0.0.0.0:8000
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *371 send() failed (111: Connection refused), context: ngx.timer, client: 172.19.0.1, server: 0.0.0.0:8000
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *371 [kong] statsd_logger.lua:76 failed to send data to localhost:8125: connection refused, context: ngx.timer, client: 172.19.0.1, server: 0.0.0.0:8000
Sorry @Rishi_Anand, I haven’t had a chance to reproduce this. My initial inclination is that there is something wrong with your go binary itself. While debugging my issues i was manually running the binary inside the alpine container to see if it would work, i would recommend trying that and see how it goes.
It would be strange though, considering my dockerfile has all versions pinned