How to load go plugins using kong helm chart?

Hi
As far I know go plugins are binary files (.so) and they are bigger than ConfigMap/Secret size limit. So have can I load go plugin to kong in dbless mode?

For now, I’ve added sidecar that is coping go plugin file to kong container. But I’ve a problem with enabling it.
This is my vaules.yaml for kong helm chart.

` plugins:

   configMaps:
       - pluginName: transformer-http-go-plugin
          name: <what should be here?>

`

  1. Please create Docker images and host it in your container registry. I’d recommend against copying it at runtime using initcontainers or sidecars.
  2. With Go plugins, all you need is to set the correct environment variable KONG_PLUGINS in this case to include your plugin name:
env:
  plugins: bundled, my-go-plugin-name

This bums me out… I don’t really want to build a custom container just to add a single file (mostly because i don’t want to spin up the accompanying ci/cd and then host that container in some registry). Is there anything wrong with loading the golang library file similar to how we mount the lua files via configmaps into the container???

Am I missing something?

Thanks,
Ben

One needs to compile go and then put that inside the container.
This feature is like pealing an onion so we decided to not add in. I do understand that there is some other overhead.

I’d be open to solving problem with this solution as well but I don’t know of anyway to actually solve it.

Yeah, i think the issue of also needing to compile the go pluginserver adds an additional layer of complexity which complicates the “just using configmaps” method that the lua plugins use. I will continue to tinker around, but I suspect you are right, and the best way is just to make a custom container.

Thanks,
Ben

my attempt at creating a docker-file and adding my plugin:

FROM golang:1.14.0 as go-builder

# Build golang plugin(s)
WORKDIR /build
COPY ./custom-go-plugins ./
WORKDIR /build/go-hello/src
RUN go get -insecure ./...
RUN go build -buildmode=plugin .

# Build golang pdk
RUN go get github.com/Kong/go-pluginserver


FROM kong:2.0
WORKDIR /custom-go-plugins
# Add golang app
COPY --from=go-builder --chown=kong:root /build/go-hello/src/go-hello.so /custom-go-plugins/
# Add the pluginserver
ENV goPath=/go
COPY --from=go-builder --chown=kong:root ${goPath}/bin/go-pluginserver /usr/local/bin/

However, when i try to run the go-pluginserver inside the container i get a “no such file or directory error” I was suspicious that this error was actually the RPC socket file “/usr/local/kong/go_pluginserver.sock” but even if i manually make the file, i get the same error… So now i am suspicious I built this the wrong way, or I am missing some golang dependency… even though everything should be statically compiled since its go. :thinking:

docker run -it bclouser/kong:2.0 /bin/bash

bash-5.0$ ls -lh /usr/local/bin/
total 14M    
-rwxr-xr-x    1 kong     root       13.9M May 13 04:39 go-pluginserver
-rwxr-xr-x    1 1000     1000         451 Apr 24 16:27 json2lua
-rwxr-xr-x    1 kong     root         238 Apr 24 16:27 kong
-rwxr-xr-x    1 1000     1000         450 Apr 24 16:27 lapis
-rwxr-xr-x    1 1000     1000         451 Apr 24 16:27 lua2json
-rwxr-xr-x    1 1000     1000        1.4K Apr 24 16:26 luarocks
-rwxr-xr-x    1 1000     1000         964 Apr 24 16:26 luarocks-admin
bash-5.0$ 
bash-5.0$ /usr/local/bin/go-pluginserver 
bash: /usr/local/bin/go-pluginserver: No such file or directory

The error in kong which led me to try running the pluginserver manually looks like this:

waiting for db
Error: /usr/local/share/lua/5.1/kong/cmd/start.lua:64: sh: /usr/local/bin/go-pluginserver: not found
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/MessagePack.lua:813: missing bytes

Fishing for advice :fishing_pole_and_fish:

My knowledge in this area is sparse. @hutchic could you help out here please?

@thatbenguy,

Note that your go-builder stage uses a Debian base image, while kong:2.0 is based on Alpine – as such, the binary you generate isn’t valid for the target system and you get that error message. Try using golang:1.14.0-alpine as Go builder and please let us know how it goes!

Well spotted @salazar! That fixed my issue with the pluginserver.

I do think i hit another issue with using both lua plugins and golang plugins side-by-side

waiting for db
Error: /usr/local/share/lua/5.1/kong/cmd/start.lua:64: 2020/05/13 23:31:06 failed to open plugin go-hello: plugin: not implemented
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:455: bad argument #1 to 'ipairs' (table expected, got nil)

I did have to add a hack to the ‘helpers.tpl’ file in order to get the ‘go-hello’ plugin added to the list of plugins, so I will continue tinkering around to see if this error is my fault, but the kong manifest output produced from a dry-run looks pretty good to me - the KONG_PLUGINS env block for the init-container and the proxy config looks like this:

- name: KONG_PLUGINS
          value: "bundled,go-hello,kong-plugin-myheader,uuid-filter,jwt-extract-uuid,jwt-claims-headers,uuid-req-transformer,uuid-uri-rewrite" 

NOTE: go-hello is the only golang plugin, all the other plugins are lua ones mounted in with configmaps

Ah, ok, i figured it out. I was building the plugins without CGO, since in order to have cgo i had to apk add build-base and I assumed I didn’t need it… A few people belly-aching about it here: https://github.com/docker-library/golang/issues/153
Since go’s plugin package requires C functions, without CGO my plugin was probably causing a seg-fault or some kind of error inside the plugin.open() function which is why i saw failed to open plugin go-hello: plugin: not implemented

Things are working as expected now. Here is the final docker-file for building this:

FROM golang:1.14.2-alpine as go-builder

# Because the plugin package needs cgo (https://golang.org/pkg/plugin/) 
# You can see some C code here: https://golang.org/src/plugin/plugin_dlopen.go
RUN apk add --no-cache build-base

# Build golang plugin(s)
WORKDIR /build
COPY ./custom-go-plugins ./
WORKDIR /build/go-hello/src
RUN go get -insecure ./...
RUN go build -buildmode=plugin .

# Build golang pluginserver
RUN go get github.com/Kong/go-pluginserver@v0.3.1


FROM kong:2.0
# Add golang app
COPY --from=go-builder --chown=kong:root /build/go-hello/src/go-hello.so /custom-go-plugins/
# Add the pluginserver
ENV goPath=/go
COPY --from=go-builder --chown=kong:root ${goPath}/bin/go-pluginserver /usr/local/bin/

And here is the patch I made to _helpers.tpl to allow adding golang plugins alongside my config-mapped LUA plugins:

--- a/kong/kong-1.5.0/templates/_helpers.tpl
+++ b/kong/kong-1.5.0/templates/_helpers.tpl
@@ -296,6 +296,9 @@ The name of the service used for the ingress controller's validation webhook
 
 {{- define "kong.plugins" -}}
 {{ $myList := list "bundled" }}
+{{- range .Values.plugins.goPlugins -}}
+{{- $myList = append $myList .pluginName -}}
+{{- end -}}
 {{- range .Values.plugins.configMaps -}}
 {{- $myList = append $myList .pluginName -}}
 {{- end -}}

So my values.yaml file has a plugins block like this:

plugins: 
  configMaps:
  - name: kong-plugin-myheader
    pluginName: kong-plugin-myheader
  - name: kong-plugin-uuid-filter
    pluginName: uuid-filter
  - name: kong-plugin-jwt-extract-uuid
    pluginName: jwt-extract-uuid
  - name: kong-plugin-jwt-claims-headers
    pluginName: jwt-claims-headers
  - name: kong-plugin-uuid-req-transformer
    pluginName: uuid-req-transformer
  - name: kong-plugin-uuid-uri-rewrite
    pluginName: uuid-uri-rewrite
  goPlugins:
  - pluginName: go-hello

Happy to submit some pull-requests… I am just not sure where, or if this is how the maintainers want to do things going forward. Let me know.
Cheers!

I am trying to follow your docker file only. Trying with go-hello plugin.
Able to start the kong successfully, but while executing the plugin, getting error. Follwing are the logs -

kong_1 | 2020/08/26 18:12:33 [crit] 26#0: *368 connect() to unix:/usr/local/kong/go_pluginserver.sock failed (2: No such file or directory), client: 172.19.0.1, server: kong, request: “GET /route1 HTTP/1.1”, host: “localhost:8000”
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *368 [kong] go.lua:140 [go-hello] trying to connect: no such file or directory, client: 172.19.0.1, server: kong, request: “GET /route1 HTTP/1.1”, host: “localhost:8000”
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *368 [kong] go.lua:429 [go-hello] starting instance: no such file or directory, client: 172.19.0.1, server: kong, request: “GET /route1 HTTP/1.1”, host: “localhost:8000”
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *368 lua coroutine: runtime error: /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:432: no such file or directory
kong_1 | stack traceback:
kong_1 | coroutine 0:
kong_1 | [C]: in function ‘error’
kong_1 | /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:432: in function ‘get_instance’
kong_1 | /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:498: in function </usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:497>
kong_1 | coroutine 1:
kong_1 | [C]: in function ‘resume’
kong_1 | coroutine.wrap:21: in function coroutine.wrap:21
kong_1 | /usr/local/share/lua/5.1/kong/init.lua:839: in function ‘access’
kong_1 | access_by_lua(nginx-kong.conf:91):2: in main chunk, client: 172.19.0.1, server: kong, request: “GET /route1 HTTP/1.1”, host: “localhost:8000”
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *368 [kong] init.lua:841 [go-hello] /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:432: no such file or directory, client: 172.19.0.1, server: kong, request: “GET /route1 HTTP/1.1”, host: “localhost:8000”
kong_1 | {“timestamp” : “26/Aug/2020:18:12:33 +0000”,“client_ip” : “172.19.0.1”,“real_ip”: “”,“host”: “localhost:8000”,“request” : {“method”: “GET”,“uri” : “/route1”,“request_length”: “647”,“x-siemens-correlation-id” : “”},“upstream” : “”,“response”: {“headers” : {“date” : “”,“X-Siemens-meter-data” : “”},“status” : 500,“size” : 42},“latencies” : {“request” : 0.011,“proxy” : “”},“rate limit” : 0,“connection_requests” : 1}
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *371 send() failed (111: Connection refused), context: ngx.timer, client: 172.19.0.1, server: 0.0.0.0:8000
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *371 [kong] statsd_logger.lua:76 failed to send data to localhost:8125: connection refused, context: ngx.timer, client: 172.19.0.1, server: 0.0.0.0:8000
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *371 send() failed (111: Connection refused), context: ngx.timer, client: 172.19.0.1, server: 0.0.0.0:8000
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *371 [kong] statsd_logger.lua:76 failed to send data to localhost:8125: connection refused, context: ngx.timer, client: 172.19.0.1, server: 0.0.0.0:8000
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *371 send() failed (111: Connection refused), context: ngx.timer, client: 172.19.0.1, server: 0.0.0.0:8000
kong_1 | 2020/08/26 18:12:33 [error] 26#0: *371 [kong] statsd_logger.lua:76 failed to send data to localhost:8125: connection refused, context: ngx.timer, client: 172.19.0.1, server: 0.0.0.0:8000

Sorry @Rishi_Anand, I haven’t had a chance to reproduce this. My initial inclination is that there is something wrong with your go binary itself. While debugging my issues i was manually running the binary inside the alpine container to see if it would work, i would recommend trying that and see how it goes.
It would be strange though, considering my dockerfile has all versions pinned :thinking:

Best of luck!