How do I use "Custom Fields by Lua" for `file-log` plugin?

Questions
1

Hello, I’m using Kong Gateway OSS (2.3.x). I’d like to customize the contents logged by the file-log plugin.

I found the following on its docs page: “The custom_fields_by_lua configuration allows for the dynamic modification of log fields using Lua code.”

What I tried is the following based on the admin-api

curl -v \
  -X POST \
  --data 'name=file-log' \
  --data 'config.path=/tmp/file.log' \
  --data 'config.custom_fields_by_lua.route="return nil"' \
  http://localhost:8001/plugins

and I receive “schema violation (config.custom_fields_by_lua: unknown field)”. Does anyone know how to use custom_fields_by_lua? Thanks!

2

This feature is available in 2.4.x only.

3

So by reading source code, I figured it out too. And I could copy the plugin to the instance I’m running, it does work. I think I got confused by the file-log plugin page which says:

Version Compatibility:
KONG GATEWAY (ENTERPRISE)
 2.4.x
 2.3.x
 2.2.x
 2.1.x
 1.5.x
 1.3-x
 0.36-x
KONG GATEWAY (OSS)
 2.4.x
 2.3.x
 2.2.x
 2.1.x
 2.0.x
 1.5.x
 1.4.x
 1.3.x
 1.2.x
 1.1.x
 1.0.x
 0.14.x
 0.13.x
 0.12.x
 0.11.x
 0.10.x
 0.9.x
 0.8.x
 0.7.x
 0.6.x
 0.5.x
 0.4.x
 0.3.x

I just hope the documentation says a little more accurately. Maybe like, This plugin works with the following versions, but the latest added feature is only shipped from 2.4. or something.

4

I’m using Kong 3.3.0 dockerized container. I was able to turn off various fields and leave just the ones I needed by this:

curl -X POST http://localhost:8001/plugins \
 --header "Content-Type: application/json" \
 --data '{
   "name": "file-log",
   "config": {
     "path": "/tmp/file.log",
     "reopen": true,
     "custom_fields_by_lua": {
       "client_ip": "return nil",
       "route": "return nil",
       "upstream_status": "return nil",
       "tries": "return nil",
       "service": "return nil",
       "response": "return nil",
       "upstream_uri": "return nil",
       "authenticated_entity": "return nil",
       "consumer.created_at": "return nil",
       "consumer.updated_at": "return nil",
       "consumer.tags": "return nil",
       "consumer.id": "return nil",
       "consumer.custom_id": "return nil",
       "latencies.request": "return nil",
       "request.querystring": "return nil",
       "request.method": "return nil",
       "request.size": "return nil",
       "request.url": "return nil",
       "request.uri": "return nil",
       "request.headers.connection": "return nil",
       "request.headers.x-consumer-username": "return nil",
       "request.headers.x-consumer-id": "return nil",
       "request.headers.x-real-ip": "return nil",
       "request.headers.postman-token": "return nil",
       "request.headers.host": "return nil",
       "request.headers.user-agent": "return nil",
       "request.headers.accept": "return nil",
       "request.headers.x-credential-identifier": "return nil",
       "request.headers.x-forwarded-proto": "return nil",
       "request.headers.x-consumer-custom-id": "return nil",
       "request.headers.accept-encoding": "return nil"
     }
   }
 }'

I don’t know how to synthesize new fields, though. Made a few basic attempts and it didn’t work out; anyways whatever I wanted was already included by the default plugin; I just had to get rid of all these extra params.

The output is a compact one-line json having the ip address, the username, unix timestamp, and some other things. And I prefer this json request body format for the api calls to Kong instead of the multipart form one - json is easier to code in as well.

5

We do it on our side (a colleague build the base logic) with a custom loa code snipped
(We restructure the log output and pack it in a substructure with the root key “kong”)
It is embedded in helm … golang templating syntax, that we can influence the log structure base on deployment variables)

But i think it can be understood what idea is behind

  kong.yml: |
    _format_version: "1.1"

    plugins:
{{- if .Values.plugins.file_log.enabled }}
      - name: file-log
        config:
          path: {{ .Values.plugins.file_log.path }}
          custom_fields_by_lua:
            kong: |
              local set_serialize_value = kong.log.set_serialize_value
  {{- if not .Values.plugins.file_log.custom_fields.log_request_headers }}
              set_serialize_value("request.headers", nil)
  {{- end }}
  {{- if not .Values.plugins.file_log.custom_fields.log_response_headers }}
              set_serialize_value("response.headers", nil)
  {{- end }}
  {{- if .Values.plugins.file_log.custom_fields.log_client_s_dn }}
              set_serialize_value("request.tls.client_s_dn", ngx.var["ssl_client_s_dn"])
  {{- end }}
              local log_data = kong.log.serialize()
              for key, expression in pairs(log_data) do
                set_serialize_value(key, nil)
              end
              return log_data
{{- end }}