As the request came into the api server, we are unable to track client’s ip. The IP we got from the request itself is stating Kong’s ip instead of client . Is there a way to forward the client’s ip to API
Hi,
This is a typical behavior/drawback of many reverse-proxies and load-balancers. Luckily for you, and as one would expect, there exist ways to circumvent them!
Retrieving the client IP in your API
First and foremost, Kong will set the X-Forwarded-For header for you since 0.11.0. You can use its value in your upstream services to retrieve the client IP. This is, I believe, the answer you are looking for.
But let’s take a deeper dive here, because there are other important areas around this topic to cover before implementing Kong in a production-ready environment.
When Kong is itself behind a load-balancer
Now, what to do when Kong’s client is a previous load-balancer/reverse-proxy hop in the HTTP chain?
The most obvious way to address this is to rely on the ngx_http_realip_module module from NGINX which is bundled in Kong’s official distribution packages. This module properly parses the X-Forwarded-For header according to rules you can configure yourself.
Kong exposes various configuration properties (trusted_ips, real_ip_header, and real_ip_recursive) that abstract the NGINX module’s directives of the same name to help you configure it in such a way so that your upstream services can retrieve the client’s address.
If your Kong instance is properly configured, it will be able to retrieve the client IP, as well as each individual IP from every previous HTTP hop between the client and itself. It will then add its own IP to the X-Forwarded-For header, and proxy the request to your upstream service. Your service is thus able to retrieve the same information as well.
Retrieving the client IP in plugins
The ngx_http_realip_module will update the $remote_addr NGINX variable to contain the client IP (granting it is correctly configured and the request comes from a trusted IP). You can access this variable in Lua from ngx.var.remote_addr.
Beware, for existing plugins already access this value like so, such as the rate-limiting plugin. This means, if Kong/ngx_http_realip_module is badly configured, this plugin will rate-limit based on a previous load-balancer’s IP address, instead of each individual client’s IP address, leading to a drastically different behavior than expected!
Other client information transferred by Kong
Kong will also set other X-Forwarded-* headers:
-
X-Forwarded-Proto, containing the protocol used by the client (httporhttps). -
X-Forwarded-Host, containing the originalHostheader sent by the client. -
X-Forwarded-Port, containing the original port against which the client initially connected to.
If the request comes from a non-trusted IP (likely, directly from a client), Kong will set those values itself. However, if the request comes from a trusted IP and those headers are present, Kong will simply forward them upstream, untouched.
Support for PROXY protocol
Rather specific to HAProxy and ELBs, this protocol is natively supported by the ngx_http_realip_module’s real_ip_header directive. The Kong equivalent property of the same name can also receive the proxy_protocol value. This is documented in the Kong configuration file and reference.
Read more
- You can find more information about those headers and Kong’s proxy behavior in the Proxying Guide section dedicated to upstream requests.
- See the code which relates to topics explored here for yourself: kong/core/handler.lua#L408-L442.
- Read on using the PROXY protocol with NGINX.
9 Likes
Hi @thibaultcha we have seen in version 0.10.3 behind F5 LB and using ip-restriction plugin, if x-forwarded header is spoofed it gets by ip-filtering in kong. If i understand your post right starting version 0.11 this isnt case correct?
IP Restriction plugin uses ngx.var.binary_remote_addr and that variable is also update by the realip module. Before 0.11 the Kong trusted X-Forwarded-For header from any source (by default). From 0.11 Kong does not by default trust that header from any source. And that is a correct way to do by default. And it will work right when clients are not behind LB. When you add LB before Kong, you need to configure at least the trusted_ips for X-Forwarded-For header to be trusted. Also please make it sure that it is your LB that adds the X-Forwarded-For header.
1 Like
Thanks @bungle
Will give it a try
Running docker 0.13.1 in a container.
/ # grep real_ip /usr/local/kong/nginx-kong.conf
real_ip_header X-Forwarded-For;
set the “ip-restriction” plugin :
$ curl $KONG_ADMIN_ENDPOINT/services/devapi.internal/plugins
{“total”:1,“data”:[{“created_at”:1529307076081,“config”:{“whitelist”:[“10.90.0.0/16”]},“id”:“3b24e2e1-b724-4c42-afb8-5ba99ed43072”,“service_id”:“ea56be3c-499c-400d-a5bc-72350bf604ae”,“name”:“ip-restriction”,“enabled”:true}]}
making request :
curl http://path-to-kong-via-elb/v1/status
getting back the response from that endpoint when I expected to be “blocked”.
log via log plugin:
{“latencies”:{“request”:103,“kong”:100,“proxy”:3},“service”:{“host”:“upstream.devapi”,“created_at”:1529300546,“connect_timeout”:60000,“id”:“ea56be3c-499c-400d-a5bc-72350bf604ae”,“protocol”:“http”,“name”:“devapi.internal”,“read_timeout”:60000,“port”:80,“path”:null,“updated_at”:1529300546,“retries”:5,“write_timeout”:60000},“request”:{“querystring”:{},“size”:“215”,“uri”:"/v1/status",“url”:“http://path-to-kong-via-elb:8000/v1/status",“headers”:{“host”:“path-to-kong-via-elb”,“x-forwarded-for”:“108.185.160.70”,“user-agent”:“curl/7.54.0”,“accept”:"/”,“connection”:“keep-alive”,“x-forwarded-proto”:“http”,“x-forwarded-port”:“80”},“method”:“GET”},“tries”:[{“balancer_latency”:0,“port”:8888,“ip”:“10.90.2.80”}],“client_ip”:“10.90.4.108”,“api”:{},“upstream_uri”:"/v1/status",“response”:{“headers”:{“content-type”:“application/json”,“date”:“Mon, 18 Jun 2018 08:57:32 GMT”,“connection”:“close”,“tt_log_http_verb”:“GET”,“tt_log_uri_name”:"/api/status",“content-length”:“452”,“via”:“kong/0.13.1”,“server”:“Cowboy”,“x-kong-proxy-latency”:“100”,“x-kong-upstream-latency”:“3”},“status”:200,“size”:“819”},“route”:{“created_at”:1529300556,“strip_path”:false,“hosts”:null,“preserve_host”:true,“regex_priority”:0,“id”:“c15a9472-09dc-4bc0-ac10-15127bc19eb2”,“paths”:["/v1/status"],“service”:{“id”:“ea56be3c-499c-400d-a5bc-72350bf604ae”},“updated_at”:1529300556,“protocols”:[“http”,“https”],“methods”:null},“started_at”:1529312253145}
you’ll notice in the log that x-forwarded-for has the value “108.185.160.70” but the request still goes through.
Is something misconfigured or what? or is something else missing?
1 Like
@nitzan.harel have you configured this as well (it looks like you have not, then we don’t trust that header)?
Hi folks , How to properly configure kong behind elb , I want to use ip-restriction plugin.
Current Setup:
INTERNET => [ ELB ] => kong => [[ our service ]]
ip restriction is not working at all. Can any one please give me step wise instructions to properly setup the kong behind ELB with IP restriction enabled
2 Likes
IP restriction plugin by default uses X-Real-IP header as the client IP. But with ELB, your client IP is in X-Forwarded-For header.
As suggested by the @bungle and @nitzan.harel , you would need to set following configs in order for the plugin to work with ELB.
trusted_ips=0.0.0.0/0,::/0
real_ip_header=X-Forwarded-For
I recently figured this out, and just tested it with our setup.
1 Like
Well Thank you so much nikhil but it fixed , some error in config i did without paying heed to documentation. yes, above config made it work perfectly.
AFAIK ELB also supports proxy_protocol if you want to try that.
1 Like
May you can look into this link: https://kubernetes.io/docs/tutorials/services/source-ip/
I have deployed my kong as a nodePort service, but Kubernetes default to use DNAT for nodePort service, I have followed the link above to changed spec.externalTrafficPolicy to Local, then the services which behind kong can get real client IP perfectly
I’m having a similar issue.
I have an nginx in front of Kong (under 10.250.3.2) and I setup Kong to trust this IP.
My nginx have the following config:
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Connection "";
When I put the office IP address as whitelist it doesn’t work but when I use 10.250.3.2 in the ip restriction whitelist, it does.
What am I doing wrong?
Kong is running within docker and I have the following config on it:
KONG_TRUSTED_IPS: 0.0.0.0/0,::0
KONG_REAL_IP_RECURSIVE: on
KONG_REAL_IP_HEADER: X-Forwarded-For
1 Like
I am using Kong 0.14.0
Adding in the trusted IPs did the trick for me. I had the other 2 set but wasn’t doing what I wanted, until I added the trusted IP section.
Hello All,
I have the almost the same issue. It passes through Load balancer and Akamai. However, the remote_addr that we are seeing is the Akamai IP and not the client IP.
We have these environment variables configured.
KONG_TRUSTED_IPS: 0.0.0.0/0,::0
KONG_REAL_IP_RECURSIVE: on
KONG_REAL_IP_HEADER: X-Forwarded-For
Is here who knows what might be missing in our config?
Does your “Load balancer and Akamai” both honor and support the X-Forwarded-For header along the path? Because I use the same exact settings as you through an F5 loadbalancer and it works perfectly when the F5 sets the X-Forwarded-For.
Yes Akamai supports X-Forwarder-For as well as Load Balancer because it’s AWS ELB.
Hi!
This is my configuration, on kong:0.14.1-centos
KONG_TRUSTED_IPS: 0.0.0.0/0,::/0
KONG_REAL_IP_HEADER: X-Forwarded-For
KONG_REAL_IP_RECURSIVE: on
$ k get po -n kong -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
kong-9fbfd678b-4lnbw 1/1 Running 0 11d 10.244.189.117 worker2 <none>
To test my configuration, I deploy k8s.gcr.io/echoserver:1.10 image, next step is to create a service + an ingress (http://linux.rocks.com) to that pod.
usuario@usuario-HP-280-G2-SFF:~ $ curl http://linux.rocks.com
Hostname: nginx-5bc49ccb64-jslsh
Pod Information:
-no pod information available-
Server values:
server_version=nginx: 1.13.3 - lua: 10008
Request Information:
client_address=10.244.189.117
I expect to get the real client ip on client_address instead of the ip of the kong-proxy pod.
If I look inside the k8s.gcr.io/echoserver:1.10 pod this is the variable to get the client_address
Request Information:
client_address={{ngx.var.remote_addr}}
Any suggestion is welcome. Thanks!