RFC: Kong native Declarative Config format

#1

Hi everyone,

We are in the process of designing a native declarative configuration format to be used in Kong.
Declarative config — specifying your full configuration in a declarative way, rather than an imperative sequence of Admin API operations — has been a desire from the community for a while. A number of features that we want to eventually have depend on having a way to specify a full Kong configuration: defining the declarative config format is the foundational step to get there.

Declarative config has been on our radar for a while, and our first stab at it was decK, an external declarative config tool. With the experience gained with this project, we’ve been working on the idea of a native declarative config format. We’d like to share with the you all our current work-in-progress design, in order to gather community feedback.

Kong Declarative Config Format (latest update 2019-02-20)

The format will be presented below in YAML, but it will be supported in JSON as well. At this point we do not have a formal specification, but we will present the intended feature set through a complete commented example:

# Metadata fields start with an underscore (_)
# Fields that do not start with an underscore represent Kong entities and attributes

# Matches Kong minimum version that supports the format
_format_version: "1.1"

# YAML supports comments, but JSON doesn't. Our format
# supports the "_comment" field in the top-level and within entities
_comment: "general comment about the file"

# The top-level and entities also support the "_ignore" field,
# which keeps an array of entries that are ignored by Kong.
# You can use it to store your additional metadata.
_ignore:
- my_own_data: 123
- foo: bar
  kong_ignores: "all of this"

# Each Kong entity (core entity or custom entity introduced by a plugin)
# can be listed in the top-level as an array of objects:
services:
- name: my-service
  url: http://mockbin.org
  # Entities will be able to store tags as metadata
  tags: [desktop, example]
  # Entities that have a foreign-key relationship can be nested:
  routes:
  - name: example-route
    # Each entity has its own, optional, set of tags
    tags: [desktop, something_else]
    hosts:
    - example.test
    # Plugins applied to example-route:
    plugins:
    - name: basic-auth
  # Plugins applied to my-service:
  plugins:
  - name: key-auth

routes:
- name: my-route
  # Relationships can also be specified between top-level entities,
  # either by name or by id
  service: my-service
  hosts: ["hello.com"]

consumers:
- username: Frodo
  # Custom entities from plugin can also be specified
  # If they specify a foreign-key relationshp, they can also be nested
  keyauth_credentials:
  - key: my-key
  plugins:
  - name: rate-limiting
    _comment: "these are default rate-limits for user Frodo"
    config:
      second: 5
      hour: 10000

# When an entity has multiple foreign-key relationships
# (e.g. a plugin matching on both consumer and service)
# it must be specified as a top-level entity, and not through
# nesting.
plugins:
- name: rate-limiting
  consumer: Frodo
  service: my-service
  _comment: "Frodo is extra limited when using my-service"
  config:
    hour: 100
  # tags are for your organization only and have no meaning for Kong:
  tags:
  - extra_limits
  - my_tag 

We are trying to keep a good balance between functionality an simplicity. Please let us know your impressions!

3 Likes

Docker compose - load configuration
Kong 1.1.0rc2 available for testing
Kong 1.1.0rc1 available for testing
#2

Seems fine to me. As with no db I imagine certain plugins like OAuth2 with client credentials flow through Kong would be unsupported and only a local rate limiting policy applied and things of that nature but overall it provides a solid way to cut out the db.

Other aspects I think will be doing it in a manner that does not require an actual redeploy/restart and parse of a local yaml file sitting on the Kong node itself, but rather can each node point to a schema file hosted remotely at a central location possibly and do some sort of kong rebuild -f https://myrepo.com/mykongrepo/kongconfig.yaml at runtime with no downtime? And then just run that cmd on every declarative kong node we run to keep them in sync? Ofc a local file should be reasonable too if thats how a user wants to do it :slight_smile: , but I would prefer one remote yaml file hosted elsewhere and say my 6 kong nodes all reference that one yaml.

Other random features floating in my head would be a way to maybe encrypt/decrypt the kong config file for some potential use case I haven’t fully vetted(share the file/keep it visible in public repo but not expose secrets stored in plaintext maybe? idk, maybe just add a layer of encryption on just the secrets in a way only Kong cluster “declarative” nodes would be able to decipher.).

If I have any other ideas i will drop them in place. I probably, won’t be leveraging this flow anytime soon due to being unable to support oauth2 cc by nature, but I do like the idea of it! Not being tied to a db and latency/connections held around that would provide a large amount of stability and not cut down too many features.

0 Likes

#3

@hisham can we provide an example of declaring an entity with a long, complex string value (e.g., a certificates object)?

0 Likes

#4

Very cool. Wonderful to see this WIP Draft!

Curious what is the underlying plan for how this config is used by Kong:

Will it be read at start-up, and the config loaded into the backing datastore like API-based configuration? In other words, can this declarative config be mixed with API-based config, or are they exclusive to one another?

0 Likes

#5

Hi Hisham,

Wondering if more than one file should be used. This helps to keep environment related details away from route definitions. Something like this:

  • consumer.yml
  • route.yml
  • service.yml

A single file containing all configurations is harder to manage.

We have developed an internal declarative configuration tool for Kong along similar lines which is working well so far.

0 Likes

#6

This is very similar to the format we came up with for kongverge, except it’s in YAML :smiley:

0 Likes

#7

For those who use k8s ingress for declarative configuration already, would this proposal dovetail with that work at all?

There are plenty of situations where a user will want to test the same set of routes and services as they have in k8s in a local development environment. This RFC would allow them to create that local dev environment declaratively. However, they’ll end up with the same rules defined in k8s ingress definitions and in this RFC’s config format, won’t they. Is there a way we could avoid the duplication?

0 Likes

#8

Yes, that’s the idea. For rate-limiting, one can also use the redis backend (effectively having a separate datastore just for that).

As for the usage workflow, in the first iteration we’re planning to provide an endpoint for (re-)uploading the configuration file, as well as a kong.conf option for specifying the file at startup (which can be overriden via env vars as usual).

Our initial plan is for this declarative file format to be used in both database-backend and DB-less scenarios:

  • when using a database, the file can be used for bulk import (essentially every item gets upserted on top of the current configuration)
  • when using db-less, the file is a truly declarative configuration: i.e. it represents the entire state of the world for Kong, so pushing a new one means the old configuration is discarded and replaced by the new one entirely.

We do have plans for that (and it’s helpful to hear that it’s a desired feature!) but the first release will probably be a single file.

We plan supporting both YAML and JSON; the conversion from one to the other is the one you expect (the _comment metadata field was added for the sake of JSON).

We will certainly look into this. We are constantly making steps to have the K8s experience for Kong to be as seamless as possible.

1 Like

#9

Sure. YAML provides syntax for that; this is a part of a generated config produced by our test suite testing mesh-related functionality using declarative config:

cluster_ca:
- pk: true
  key: |
    -----BEGIN PRIVATE KEY-----
    MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDl1DUYwTaaAyeX
    EuEH19HzN1HzrDNSKDvTVDgLpTAg0rRSn9FMNZ6iEHOSEmmG5xd15Wh2iGoBvKKX
    6VSgPN9cnRYXL8pi9GGU+1iLMiwCJeF2jwO7st38DKlQpBshvKKypkAZeWidydzm
    F7GscMWDvSIKHnz3a1OGw6Ko2Qjp92GJ3aDp+LypbJrmODBXjD0xixdXCLZXsAWr
    gFFxCQJBXd7iHWr6pgQcUukqglSqp07Hoc8kjENwQ8gFmQawQUQEv6v1xg0fWSbG
    UTtzY1spqZiTeIIt0HTKEr8oHW2jJ+pVwboCFVJyJ4yzDYZwYbQLCwFN+lkVwXVg
    WniSHThpAgMBAAECggEAe7RT/AYLaKxoA9Xij8EnvXy6PKkkWoVTZjU/kW0RUal3
    670f/6AgyWSHgEkuybDbHNck+iAExviMn620oicBpqzathJAIjEd0P3Us9+GdoWD
    LLhfrwjgmQkLnV8nR2psPDJl8BA8Min0H1kDhiiQ9462orG0W8xGAhSNGcrwFFXQ
    zMWBHz7eIVqdcdwvYRh690RPNm8CIJ5QIRyH+Qj+GGJUiSv1LOfsYjjmemn1BBYe
    s943MW47yX+ZzFjjeuFY+onlkfxzBVwOCGOc9o1LhINv2uTqlGP8mQU2dlWmZN7n
    MU+m6bTF7GThvAT7R0FZgw8LtCSqxxZF0hJckuvugQKBgQD5rOX1fvs/YjlyMwfj
    DqhYZ8M2h8Ie76jkwtHvW9KTliiIE6VnJ/499GKLMTuEHj3QLua7+RrHCNt5itoi
    AHekQrh2PA1T0cYH6mwujgZYCA8xNc7I3p35jCQ7PzsDnQbMQeJd9WPBGyN4jVf/
    3rw4oyKTCdqsapOd4cwVw0md0QKBgQDrppu8P9TRIltHUnlLMU4cTuCDWjx6XZEi
    b+anxS/gx358Uqu6STdxqndNtxZTY67u/W0HohqOjmmU6V7UhOfaAWvYdpWgrjY9
    Pe2tPJu4xHiTMoUsP2iVm4XTYyWRTcGj9UUN0iinpcQTca4g202XTML/3gLmZujc
    tbejimifGQKBgF0NNDM3nRn1WKWYIUNdTh6MpXOoSw/OkCqZ9TX5CdUZbIeykr2M
    aTPb5fIsahsGTlFNeY7q+zDcuzdkKYHsnKpqd2cRYIIxt1Vx59m98t1SO59Ch4vT
    dBryQPqXyT3CgysTDvOnpgLtQ14Fh5uD/rA5FISqcoY5vkRkR/SAcOpRAoGBAKqF
    paxThGzRr7usspj21M8nRM/IPHtPufZxgzdkOYEu2xibuijTUouuDybtlfpB6BSI
    zpz3cMA3za80MKsYga3/2UqKhTwA1L78UUFK2mSZx6a8qCGcwNGB+RU+abD702QR
    2CZoLYMjMP/hA2BDUaJqqbD9tpuDXH5aAMkXHx4BAoGAPmsphan5znM2OY9y2HGz
    Fc9umiWSD1v0TcL8KJX0ZRCnLgCgiiG/YM0dI6Bqjn70QaXCAcNtm1Z7eWgELW9p
    l5D+N+jRxdOlCs27jPiy4jxPsLqvZvbPlgCuCSW3BAca+C8vbufVly/U2Ap32Zun
    yGAoyZ33z5eOkewlfvEoHA0=
    -----END PRIVATE KEY-----
  cert: |
    -----BEGIN CERTIFICATE-----
    MIIDZjCCAk6gAwIBAgIRAJZjrSfUGoLGh2zJBQRZm3EwDQYJKoZIhvcNAQELBQAw
    PDE6MDgGA1UEAwwxa29uZy1jbHVzdGVyLTM5YmIwZGRkLTZlOWEtNGNiNi1hMzM2
    LTU2Y2NkYzE0NmYwNTAeFw0xOTAyMTQxNTM2MDdaFw0zOTAyMDkxNTM2MDdaMDwx
    OjA4BgNVBAMMMWtvbmctY2x1c3Rlci0zOWJiMGRkZC02ZTlhLTRjYjYtYTMzNi01
    NmNjZGMxNDZmMDUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDl1DUY
    wTaaAyeXEuEH19HzN1HzrDNSKDvTVDgLpTAg0rRSn9FMNZ6iEHOSEmmG5xd15Wh2
    iGoBvKKX6VSgPN9cnRYXL8pi9GGU+1iLMiwCJeF2jwO7st38DKlQpBshvKKypkAZ
    eWidydzmF7GscMWDvSIKHnz3a1OGw6Ko2Qjp92GJ3aDp+LypbJrmODBXjD0xixdX
    CLZXsAWrgFFxCQJBXd7iHWr6pgQcUukqglSqp07Hoc8kjENwQ8gFmQawQUQEv6v1
    xg0fWSbGUTtzY1spqZiTeIIt0HTKEr8oHW2jJ+pVwboCFVJyJ4yzDYZwYbQLCwFN
    +lkVwXVgWniSHThpAgMBAAGjYzBhMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/
    BAQDAgEGMB0GA1UdDgQWBBTI9CtdvEtVXwYue3RIkHLITyoLBzAfBgNVHSMEGDAW
    gBTI9CtdvEtVXwYue3RIkHLITyoLBzANBgkqhkiG9w0BAQsFAAOCAQEAoQt05xGY
    PZnC0GdaBnu95NgjQypHlGY+r/3e29t/E4ECb6nf29vOPwCMki8G4MEARq46mPsX
    9BJUGfw5ze4k4mScZXY0e9yogxovyQSrruAscXAmjUjscFSsBt0M+LH/1+nHivFx
    pC5gjYocIVLHYIvIYfioWqjIcIXp4o8yvQqtxj//I1i8Gp7sx6tXEEh1SBIWyChJ
    fa7DBKnDqGSJVrZecAS+FFOVsDlQ7s1Ycc+uoNsg+dAox33rIfgGbCdmWnNkoTEv
    efvuzJQmwRD9uHqZdctxw7X3h1s1eATSds0m1SJdZp66KtCCI1QCx09gDN4sfXMF
    8503vuCqhSql1w==
    -----END CERTIFICATE-----
...

Note that in this first iteration we’re not tackling the issue of securing any secrets in the config file (the format accepts in principle any core or plugin entity and configuration); at this point the user’s workflow will have to take this into consideration.

0 Likes

#10

Thank you all for your feedback! We now have a PR open implementing the first iteration of the declarative config features, aiming for Kong 1.1:

2 Likes

#11

Hi,
We are doing exactly the same thing using .net, and for us will be very useful if kong supports this by default.

One thing that we are creating on top of this is some kind of abstraction of KONG, to avoid that 1000 engineers need to know everything of Kong. The idea is a Resource concept with yaml blueprint that is transformed on kong model.

Are you thinking in create some kind on extension/abstraction to allow anyone to create their own model?

0 Likes

#12

Hi,

declarative configuration is definitely a great improvement towards infrastructure as a code. After some considerations we want to propose some further improvements or fine tuning:

  • support parsing without a running kong instance:

this would make it easier to build an automated delivery pipeline where the configuration is maintained in an e.g. git repository. A standard build server could perform a parse before a merge of a feature branch or before a deployment to kong without the need to be able to access the Admin API (which could for security reasons be forbidden for a build server which is used for checking the new configuration)

  • support for multiple configuration files:

this would enable to separate configuration of different services into different files, especially with larger configuration this can potentially increase the overview of the relevant parts for a service.

this would further allow to separate sensitive configurations for certain plugins, i.e. acl plugin.

personally I see two different approaches, either only a solution like textual import aka include or something like modules (based on services?) where the separation has semantics, too.

1 Like

#13

Another thing is nesting cross-cutting plugins where the configurations are the same and re-use by nesting on services/routes/consumers configuration

0 Likes