We are using Kong proxy 2.2.1 and kong-ingress-controller 0.9.1 on EKS. Our EKS Nodes, Kube-proxy and PODs use CGNAT CIDR range of 100.x.x.x
We are having issues with getting client’s correct IP in our services. It appears that Kong is not sending x-forwarded-for header to upstream
As a result upstream is unable to determine the client’s IP address correctly. It does however send x-forwarded-by header
Here is how the request flow for our setup looks like:
Client Browser -> AWS ALB -> Kong NLB -> Kong POD -> Service1 (Tomcat) -> Service2 (Netty)
In this example
168.2xx.xx.xxx → actual client ip
100.64.148.206 → IP of kube-proxy
100.64.96.127 → IP of Kong Pod
100.64.91.26 → IP of Service1 POD
Service 1 passes along all the “x-forwarded-*” headers to Service 2 as received.
Service 1 seems to get the correct client IP but service 2 is unable to do so. I am suspecting it must be due to missing x-forwarded-for header.
Related Kong logs
[notice] 25#0: *2364795 [kong] handler.lua:271 [my-plugin] :kong.request.get_forwarded_host:qa.mysite.com, client: 168.2xx.xx.xxx, server: kong, request: "POST /token/api/uri HTTP/1.1", host: "qa.mysite.com" [notice] 25#0: *2364795 [kong] handler.lua:275 [my-plugin] :Header: x-forwarded-for: 168.2xx.xx.xxx, client: 168.2xx.xx.xxx, server: kong, request: "POST /token/api/uri HTTP/1.1", host: "qa.mysite.com"
Related Service 1 logs
[https-jsse-nio-8443-exec-9] com.mysite.service1.filter.LoggingFilter - RemoteIP: 168.2xx.xx.xxx scheme: https URL: /token/api/uri QS: null [https-jsse-nio-8443-exec-9] com.mysite.service1.filter.LoggingFilter - x-forwarded-by: 100.64.148.206, 100.64.96.127 [https-jsse-nio-8443-exec-9] com.mysite.service1.filter.LoggingFilter - x-forwarded-proto: https [https-jsse-nio-8443-exec-9] com.mysite.service1.filter.LoggingFilter - x-forwarded-host: qa.mysite.com [https-jsse-nio-8443-exec-9] com.mysite.service1.filter.LoggingFilter - x-forwarded-port: 443 [https-jsse-nio-8443-exec-9] com.mysite.service1.filter.LoggingFilter - x-forwarded-path: /token/api/uri [https-jsse-nio-8443-exec-9] com.mysite.service1.filter.LoggingFilter - x-real-ip: 168.2xx.xx.xxx
Related Service 2 logs
[boundedElastic-8] com.mysite.service2.filter.LoggingFilter - RemoteIP: 100.64.91.26 URL: /token/api/uri QS: {} [boundedElastic-8] com.mysite.service2.filter.LoggingFilter - x-forwarded-proto: https [boundedElastic-8] com.mysite.service2.filter.LoggingFilter - x-forwarded-port: 443 [boundedElastic-8] com.mysite.service2.filter.LoggingFilter - x-forwarded-host: qa.mysite.com [boundedElastic-8] com.mysite.service2.filter.LoggingFilter - x-forwarded-by: 100.64.148.206, 100.64.96.127 [boundedElastic-8] com.mysite.service2.filter.LoggingFilter - x-forwarded-path: /token/api/uri
Our Kong deployment is configured to set following env variables
- name: KONG_TRUSTED_IPS
value: "10.0.0.0/8,100.0.0.0/8"
- name: KONG_REAL_IP_RECURSIVE
value: "on"
- name: KONG_REAL_IP_HEADER
value: "X-Forwarded-For"
What am I missing here?