Kong http2 support


#1

Hello everyone,

I’m struggling to enable http2 on kong running as an ingress controller.
Seems to me everything is setup as it should.

    spec:
      containers:
      - env:
        - name: KONG_PG_PASSWORD
          value: kong
        - name: KONG_PG_HOST
          value: kong-postgres
        - name: KONG_PROXY_ACCESS_LOG
          value: /dev/stdout
        - name: KONG_PROXY_ERROR_LOG
          value: /dev/stderr
        - name: KONG_ADMIN_LISTEN
          value: "off"
        - name: KONG_NGINX_HTTP_INCLUDE
          value: /opt/config/kong-prometheus.conf
        - name: KONG_HTTP2
          value: "on"
        - name: KONG_PROXY_LISTEN
          value: 0.0.0.0:80, 0.0.0.0:443 ssl http2

from inside the container i see that kong is started properly.

/ # ps auxwww
PID   USER     TIME  COMMAND
    1 root      0:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx -p /usr/local/kong -c nginx.conf
   25 nobody    0:02 nginx: worker process
   26 nobody    0:00 nginx: worker process
   27 nobody    0:00 nginx: worker process
   28 nobody    0:00 nginx: worker process
   29 nobody    0:00 nginx: worker process
   30 nobody    0:00 nginx: worker process
   31 nobody    0:00 nginx: worker process
   32 nobody    0:00 nginx: worker process
   42 root      0:00 /bin/sh
   57 root      0:00 /bin/sh
   65 root      0:00 ps auxwww
/ # 

/ # cat /usr/local/kong/nginx.conf 
worker_processes auto;
daemon off;

pid pids/nginx.pid;
error_log /dev/stderr notice;

worker_rlimit_nofile 1048576;

events {
    worker_connections 16384;
    multi_accept on;
}

http {
    include 'nginx-kong.conf';
}

relevant snippet from nginx-kong.conf

server {
    server_name kong;
    listen 0.0.0.0:80;
    listen 0.0.0.0:443 ssl http2;
    error_page 400 404 408 411 412 413 414 417 494 /kong_error_handler;
    error_page 500 502 503 504 /kong_error_handler;

Yet, still when testing an api with curl it returns with http/1.1

curl  --http2-v -H "apikey: REDACTED" https://api-ENDPOINT/health 
[...]
* ALPN, offering h2
* ALPN, offering http/1.1
[...]
* ALPN, server accepted to use http/1.1
[...]

When i hit the same service using a plain old nginx ingress controller it works fine.

* ALPN, server accepted to use h2
[...]
< HTTP/2 200 
< server: nginx/1.15.6
< date: Sat, 12 Jan 2019 19:57:07 GMT
< content-type: application/vnd.spring-boot.actuator.v2+json;charset=UTF-8
< access-control-allow-origin: *
< access-control-allow-methods: POST, GET, OPTIONS, DELETE
< access-control-max-age: 3600
< access-control-allow-headers: Content-Type, x-requested-with, X-Custom-Header, X-B3-TraceId, X-B3-SpanId, X-auth-token
< access-control-expose-headers: X-auth-token
< x-auth-token: fef52643-b4b6-4591-8143-106773d95f02
< strict-transport-security: max-age=15724800; includeSubDomains

I must be missing something simple


#2

You’ve not enabled http2 on the HTTP (plain-text) port:

name: KONG_PROXY_LISTEN
          value: 0.0.0.0:80, 0.0.0.0:443 ssl http2

should be

name: KONG_PROXY_LISTEN
          value: 0.0.0.0:80 http2, 0.0.0.0:443 ssl http2

#3

@hbagdi i just tried this on kong-proxy deployment, no dice.
Here’s full nginx-kong.conf

charset UTF-8;

error_log syslog:server=kong-hf.konghq.com:61828 error;

error_log /dev/stderr notice;


client_max_body_size 0;
proxy_ssl_server_name on;
underscores_in_headers on;

lua_package_path './?.lua;./?/init.lua;;;';
lua_package_cpath ';;';
lua_socket_pool_size 30;
lua_max_running_timers 4096;
lua_max_pending_timers 16384;
lua_shared_dict kong                5m;
lua_shared_dict kong_db_cache       128m;
lua_shared_dict kong_db_cache_miss 12m;
lua_shared_dict kong_locks          8m;
lua_shared_dict kong_process_events 5m;
lua_shared_dict kong_cluster_events 5m;
lua_shared_dict kong_healthchecks   5m;
lua_shared_dict kong_rate_limiting_counters 12m;
lua_socket_log_errors off;
# injected nginx_http_* directives
include /opt/config/kong-prometheus.conf;
lua_shared_dict prometheus_metrics 5m;

init_by_lua_block {
    Kong = require 'kong'
    Kong.init()   
}

init_worker_by_lua_block {
    Kong.init_worker()
}


upstream kong_upstream {
    server 0.0.0.1;
    balancer_by_lua_block {
        Kong.balancer()
    }
    keepalive 60; 
}

server {
    server_name kong;
    listen 0.0.0.0:80 http2;
    listen 0.0.0.0:443 ssl http2;
    error_page 400 404 408 411 412 413 414 417 494 /kong_error_handler;
    error_page 500 502 503 504 /kong_error_handler;

    access_log /dev/stdout;
    error_log /dev/stderr notice;
client_body_buffer_size 8k;

    ssl_certificate /usr/local/kong/ssl/kong-default.crt;
    ssl_certificate_key /usr/local/kong/ssl/kong-default.key;
    ssl_protocols TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_certificate_by_lua_block {
        Kong.ssl_certificate()
    }

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128
-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256;


    real_ip_header     X-Real-IP;
    real_ip_recursive  off;

    # injected nginx_proxy_* directives

    location / {  
        default_type                     '';

        set $ctx_ref                     '';
        set $upstream_host               '';
        set $upstream_upgrade            '';
 set $upstream_connection         '';
        set $upstream_scheme             '';
        set $upstream_uri                '';
        set $upstream_x_forwarded_for    '';
        set $upstream_x_forwarded_proto  '';
        set $upstream_x_forwarded_host   '';
        set $upstream_x_forwarded_port   '';

        rewrite_by_lua_block {
            Kong.rewrite()
        }

        access_by_lua_block {
            Kong.access()
        }

        proxy_http_version 1.1;
        proxy_set_header   Host              $upstream_host;
        proxy_set_header   Upgrade           $upstream_upgrade;
        proxy_set_header   Connection        $upstream_connection;
        proxy_set_header   X-Forwarded-For   $upstream_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $upstream_x_forwarded_proto;
        proxy_set_header   X-Forwarded-Host  $upstream_x_forwarded_host;
        proxy_set_header   X-Forwarded-Port  $upstream_x_forwarded_port;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_pass_header  Server;
        proxy_pass_header  Date;
        proxy_ssl_name     $upstream_host;
        proxy_pass         $upstream_scheme://kong_upstream$upstream_uri;

        header_filter_by_lua_block {
            Kong.header_filter()
        }
 body_filter_by_lua_block {
            Kong.body_filter()
        }

        log_by_lua_block {
            Kong.log()
        }
    }

    location = /kong_error_handler {
        internal; 
        uninitialized_variable_warn off;

        content_by_lua_block {
            Kong.handle_error()
        }

        header_filter_by_lua_block {
            Kong.header_filter()
        }

        body_filter_by_lua_block {
            Kong.body_filter()
        }

        log_by_lua_block {
            Kong.log()
        }
    }
}

kong and nginx versions:

/usr/local/kong # kong version
1.0.0
/usr/local/kong # /usr/local/openresty/nginx/sbin/nginx -v
nginx version: openresty/1.13.6.2
/usr/local/kong # /usr/local/openresty/nginx/sbin/nginx -V
nginx version: openresty/1.13.6.2
built by gcc 6.3.0 (Alpine 6.3.0) 
built with OpenSSL 1.1.1  11 Sep 2018
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -I/tmp/tmp/openssl-1.1.1/include' --add-module=../ngx_devel_kit-0.3.0 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.13 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.5 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -L/tmp/tmp/openssl-1.1.1 -Wl,-rpath,/usr/local/kong/lib' --with-pcre=/tmp/tmp/pcre-8.41 --with-pcre-jit --with-http_realip_module --with-http_ssl_module --with-http_stub_status_module --with-stream_ssl_preread_module --with-http_v2_module --with-stream --with-stream_ssl_module
/usr/local/kong # 

#4

@dcherniv

You are right. I have one fix available here:

Please let us know if that fixes your issue?