Array of strings in plugin configuration: order not respected

pamiel · July 13, 2018, 4:04pm

Dear Kong-ers

I’m currently developing a plugin where a configuration parameters (let’s call it param) is an array of strings.
I have different ways to set the configuration parameters for this plugin:

Using a single string containing the list of comma separated values. Within a curl command, it would be something like --data “config.param=value1,value2,value3,value4”
Or using the [] notation for defining item per item, as what I have seen in an example in the request-transformer plugin documentation (that is heavily using arrays of strings): --data “config.param[1]=value1” --data “config.param[2]=value2” --data “config.param[3]=value3 --data “config.param[4]=value4”

The issue is that:

When using the first method, the order to the strings within the array is kept: the order of the items presents, at the end, in the config.param Lua table accessible in the handler function of the plugin is the same as in the curl command sent to configure the plugin.
But when using the second method, the order is NOT respected: the items in the config.param Lua table are scrambled

and of course, the order to the items is (very) important for me

I dug into Kong’s code up to the Lapis routes attachment (within kong/api/init.lua), dumping the content of the config object at the earlier step as possible, to see that the order was already wrong at that level:

dump(): {
  api_name_or_id = "myApi",
  name = "myPlugin",
  ["config.param"] = {
    ["4"] = "value4",
    ["3"] = "value3"
    ["1"] = "value1",
    ["2"] = "value2",
  }
}

Then, obviously, after transformation into the kong/apis/init.lua/parse_params function, the order is still wrong:

dump(): {
  api_name_or_id = "myApi",
  config = {
    param = {
      "value4",
      "value3",
      "value1",
      "value2"
    }
  },
  name = "myPlugin"
}

and this is what I get in the handler.

So where does the scrambling happens ? Well, it looks it is inside the Lapis framework… more precisely here: https://github.com/leafo/lapis/blob/master/lapis/application.lua#L449
Tracing the json string before this line, the order within the string is still correct; tracing the Lua table after this call, the order is incorrect !
String before calling cjson call:

dump(): "{\"name\": \"myPlugin\", \"config.param[1]\": \"value1\", \"config.param[2]\": \"value2\", \"config.param[3]\": \"value3\", \"config.param[4]\": \"value4\"}"

I know that my issue is then not directly inside Kong, but I would get the advice from Kong and Lua experts:

Is the usage of “[]” in the curl/httpie commands for plugin configuration a recognized way for inputting arrays of string, or is it just a “trick” to be forbidden? Is there alternate ways to input arrays of strings, other than the two I presented above, and that I can test?
Is it normal that cjson does not respect the order of the original string using the [] notation ? Is there any way to enfore the order within cjson?

Thanks for your valuable help.

jeremyjpj0916 · July 17, 2018, 2:49am

My understanding was Lua doesn’t have a guaranteed order on table entries. One way you can do it is just string with comma separated and then string split the comma when needed. That will maintain your order(not the most elegant solution but hey works).

Tieske · July 26, 2018, 2:46pm

what seems to go wrong here is that the indices get decoded as strings. As @jeremyjpj0916 pointed out there is no order in Lua, but that only applies to the hash-part of a table. In this case the numeric indices are strings, which hence end up in the hash part, without explicit order.

your first code block shows the indices as ["4"] indicating the string values. The second block it has been decoded to a list, with numeric indices, which you can tell by the absence of the indices.

I fail to see why this could not be handled properly?

Tieske · July 27, 2018, 12:46pm

@pamiel I cannot reproduce this.

The code doing the transformation is here: https://github.com/Kong/kong/blob/master/kong/tools/utils.lua#L325-L377

It is invoked when you do a request like:

curl -X POST http://localhost:8001/services/example-service/plugins \
  --data "name=request-transformer" \
  --data "config.add.headers[1]=h1:v1" \
  --data "config.add.headers[2]=h2:v1" \
  --data "config.add.headers[3]=h3:v2" \
  --data "config.add.headers[4]=h4:v3"

when adding some debug statements:

  local function decode_array(t)
print("1", require("pl.pretty").write(t))

    local keys = {}
    local len  = 0
    for k in pairs(t) do
      len = len + 1
      local number = tonumber(k)
      if not number then
        return nil
      end
      keys[len] = number
    end
print("2", require("pl.pretty").write(keys))

    table.sort(keys)
    local new_t = {}

    for i=1,len do
      if keys[i] ~= i then
        return nil
      end
      new_t[i] = t[tostring(i)]
    end
print("3", require("pl.pretty").write(new_t))

    return new_t
  end

I get this in the logs, when doing the above curl command:

2018/07/27 12:30:51 [notice] 19244#0: *14 [lua] utils.lua:326: decode_array(): 1{
  ["2"] = "h2:v1",
  ["4"] = "h4:v3",
  ["1"] = "h1:v1",
  ["3"] = "h3:v2"
}, client: 127.0.0.1, server: kong_admin, request: "POST /services/example-service/plugins HTTP/1.1", host: "localhost:8001"
2018/07/27 12:30:51 [notice] 19244#0: *14 [lua] utils.lua:338: decode_array(): 2{
  2,
  4,
  1,
  3
}, client: 127.0.0.1, server: kong_admin, request: "POST /services/example-service/plugins HTTP/1.1", host: "localhost:8001"
2018/07/27 12:30:51 [notice] 19244#0: *14 [lua] utils.lua:349: decode_array(): 3{
  "h1:v1",
  "h2:v1",
  "h3:v2",
  "h4:v3"
}, client: 127.0.0.1, server: kong_admin, request: "POST /services/example-service/plugins HTTP/1.1", host: "localhost:8001"

Note: the array conversion bails out if the array is malformed (an entry missing for example, or also having a non-numeric key in the table)

pamiel · August 21, 2018, 1:00pm

Hello @Tieske,

Sorry for the late answer, I was 15000km far from my computer… fighting against earthquakes…

Now I’m back (and safe!), and I’m trying to put the same logs as yours…
…and I’m realizing that this piece of code has slightly changed in version 0.13.x and 0.14.x… and the tests I performed where on 0.12.3 !

So by making some copy-paste into my 0.12.3 instance (adding the decode_array function and the decode_args functions) + modifying the kong/api/init.lua/parse_params function in order to make it similar to https://github.com/Kong/kong/blob/92c87dfa5d0ace7381d877b3ce2d3b737cd6b519/kong/api/init.lua#L38):

elseif find(content_type, "application/x-www-form-urlencode", 1, true) then
   self.params = decode_args(self.params)

then I made it work: the parameters are now well sorted… but only in case the parameters are provided with a content type set to “application/x-www-form-urlencode”, i.e. in the form:

curl -X POST http://localhost:8001/apis/example/plugins --data "name=myPlugin" --data "config.param[1]=value1" --data "config.param[2]=value2" --data "config.param[3]=value3" --data "config.param[4]=value4"

When doing the same test, setting the parameters as a JSON file, with a content-type set to “application/json”, like the following:

curl -v -X POST http://localhost:8001/apis/example/plugins --header "Content-Type: application/json" --data '{"name":"myPlugin", "config.param[1]":"value1", "config.param[2]":"value2", "config.param[3]":"value3", "config.param[4]":"value4"}'

then the parameters are still scrambled, which makes sense as the call to decode_args (which calls decode_array) is only performed in case the content type is “application/x-www-form-urlencode”.

I tried to complement parse_params so that it also manages the “application/json” content-type the same way:

elseif find(content_type, "application/x-www-form-urlencode", 1, true) then
    self.params = decode_args(self.params)
elseif find(content_type, "application/json", 1, true) then
    self.params = decode_args(self.params)

and then it works for JSON input as well… but to be frank, I don’t know what could be the side effects of adding these lines for other use cases!

As a conclusion, I would say that:

Version 0.12.3 is buggy on this point as the parameters are not sorted… and I think I have no easy way to solve the issue… else than upgrading to 0.14
Despite the changes made in versions 0.13.x and 0.14.x, I suspect that the same bug is still present, but only if the parameters are provided in JSON format (I only suspect because I only made some tests on my patched 0.12.3 which behavior on this part should look like the 0.14…).

I may make a PR on the 0.14 to manage the JSON content type by adding the few lines mentioned above, but I would like to get your ‘go’ first, regarding the potential side effects of this update.

Tieske · August 28, 2018, 6:53am

@pamiel the example you gave with json doesn’t work because it is not proper json. The dots in the parameter names are separators, and hence should be nested structures. Try formatting it as proper json.

try this:

{"name": "myPlugin", "config": { "param": ["value1", "value2", "value3", "value4"]}}

pamiel · August 28, 2018, 2:57pm

Oops , sorry, you are fully right ! My JSON is obvisouly not correct…
So, running again the tests:

On v0.12.x, arrays provided in JSON format are well processed (i.e. sorted) while arrays provided in the form of a “application/x-www-form-urlencode” are not…
That’s, I suppose, the reason why some code has changed on this part in version 0.13.0 (adding the call to utils.decode_args) so that, starting from 0.13.0, arrays are now well sorted in all cases (when input as a JSON or as a “application/x-www-form-urlencode” content-type)

Thanks a lot for your help, it is now clear in my mind what I need to patch on 0.12… or (preferably) that I’m safe on this point to migrate on 0.14 !

Indeed, what generated some trouble in my mind is the fact that I’m using httpie for easily providing the JSON structure. When providing a string such as "a.b.c=foo" in the httpie command line, it is transformed into a "a.b.c": "foo" element into the JSON, and not what I expected to be: {"a": {"b": {"c": "foo"}}}.
I did not realized this because this “erroneous” JSON is indeed processed correctly by Kong (or let’s say the underlying cjson parsing layer), as if it was the expected JSON structure {"a": {"b": {"c": "foo"}}}!
So far, I did not encountered any issue as I was only providing elements containing a string, a boolean, a numerical, and even array values… but it is not working for a sorted array!
For sure, I cannot blame anyone as the original JSON is not correct (I may ask the question if such a JSON is really supposed to be parsed like that when containing element names using dots…)… and as a good JSON produces a good result!
Thanks again !

Topic		Replies	Views
Golang plugin parameters (like schema.lua)	2	568	October 12, 2020
Schema for "config" parameter for plugins Questions	0	379	June 3, 2020
Golang Plugin Read Configuration Values in Constructor Questions	0	340	January 13, 2021
Request-Transformer plugin: output of config.add.body parameter are not in order Questions	0	291	May 29, 2020
How to set priority for Go based kong custom Plugin? Questions	1	471	August 6, 2020

Array of strings in plugin configuration: order not respected

Related Topics