Client's IP address in AWS Lambda

Feature Suggestions
1

It would be awesome if we could get the Client’s IP address in lambda functions! I think we could quite easily include it in the headers section and under the current config forward_request_headers :slight_smile:

2

Hi,

You should already be able to do so if Kong is properly configured. See How to Forward Client's request IP

3

Kong is correctly configured as I have access to it in other APIs, it just doesn’t seem to be passed to the Lambda function as headers :disappointed:

Edit: Kong is setup in Proxy Protocol mode, not sure if its anything to do with that?

4

How are you connecting Kong with your Lambda function? I can easily try to debug in my end using the same flow and let you know… is it by using the Lambda plugin?

5

So heres my setup:

ELB → Proxy Protocol → Kong (CE 0.12.3) → Lambda

So a fairly straight forward setup, and yes I’m using the Lambda plugin :slight_smile:

Edit: So heres some stuff to start debugging:

Heres the nodejs lambda function:

exports.handler = function(event, context, callback) {
event.context = context;
event.env = process.env;

callback(null, event);

}

Here is the output from that function when I call in a browser context:

{
“request_body_args”: {},
“request_method”: “GET”,
“request_uri”: “/test_lambda”,
“request_headers”: {
“host”: “test.jumplead.com”,
“accept-encoding”: “gzip, deflate, br”,
“user-agent”: “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36”,
“accept”: “text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8”,
“accept-language”: “en-GB,en;q=0.9,en-US;q=0.8”,
“upgrade-insecure-requests”: “1”
},
“request_uri_args”: {},
“context”: {
“callbackWaitsForEmptyEventLoop”: true,
“logGroupName”: “/aws/lambda/test_lambda_node”,
“logStreamName”: “2018/03/19/[$LATEST]8e6327c0e7b14680a725fff59b36add9”,
“functionName”: “test_lambda_node”,
“memoryLimitInMB”: “128”,
“functionVersion”: “$LATEST”,
“invokeid”: “06d27783-2b4e-11e8-b7c5-fbde7d9ab00d”,
“awsRequestId”: “06d27783-2b4e-11e8-b7c5-fbde7d9ab00d”,
“invokedFunctionArn”: “arn:aws:lambda:eu-west-1:409744824593:function:test_lambda_node”
},
“env”: {
“NODE_ENV”: “production”,
“PATH”: “/var/lang/bin:/usr/local/bin:/usr/bin/:/bin”,
“LANG”: “en_US.UTF-8”,
“TZ”: “:UTC”,
“LD_LIBRARY_PATH”: “/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib”,
“LAMBDA_TASK_ROOT”: “/var/task”,
“LAMBDA_RUNTIME_DIR”: “/var/runtime”,
“AWS_REGION”: “eu-west-1”,
“AWS_DEFAULT_REGION”: “eu-west-1”,
“AWS_LAMBDA_LOG_GROUP_NAME”: “/aws/lambda/test_lambda_node”,
“AWS_LAMBDA_LOG_STREAM_NAME”: “2018/03/19/[$LATEST]8e6327c0e7b14680a725fff59b36add9”,
“AWS_LAMBDA_FUNCTION_NAME”: “test_lambda_node”,
“AWS_LAMBDA_FUNCTION_MEMORY_SIZE”: “128”,
“AWS_LAMBDA_FUNCTION_VERSION”: “$LATEST”,
“_AWS_XRAY_DAEMON_ADDRESS”: “169.254.79.2”,
“_AWS_XRAY_DAEMON_PORT”: “2000”,
“AWS_XRAY_DAEMON_ADDRESS”: “169.254.79.2:2000”,
“AWS_XRAY_CONTEXT_MISSING”: “LOG_ERROR”,
“_X_AMZN_TRACE_ID”: “Root=1-5aaf7233-a04dbb54b3fd6bc6d2c14140;Parent=47fa7c4827b35ab3;Sampled=0”,
“AWS_EXECUTION_ENV”: “AWS_Lambda_nodejs6.10”,
“_HANDLER”: “index.handler”,
“NODE_PATH”: “/var/runtime:/var/task:/var/runtime/node_modules”,
“AWS_ACCESS_KEY_ID”: “",
“AWS_SECRET_ACCESS_KEY”: "",
“AWS_SESSION_TOKEN”: "******”
}
}

6

Hi @Adam_Curtis

So I made my test, and here’s my result:

Lambda function:

exports.handler = (event, context, callback) => {
    
    event.context = context;
    event.env = process.env;

    console.log('EVENT', event);

    callback(null, event);
};

I’ve attached and configured the plugin into my /time API, looking into CloudWatch, I got:

2018-03-20T00:39:24.999Z	236cb636-2bd7-11e8-807e-ad685023686d	EVENT {  
   request_method:'GET',
   request_uri:'/time',
   request_headers:{  
      host:'api.<removed_my_domain>.com:8443',
      'x-consumer-username':'balexandre',
      'postman-token':'c9cb89d1-f22e-4365-83e5-d7ae48e4e89d',
      accept:'*/*',
      connection:'keep-alive',
      'x-forwarded-proto':'https',
      'x-consumer-id':'99de7e04-199a-48b4-b5cf-dcdc5a0b61df',
      'cache-control':'no-cache',
      cookie:'__cfduid=de747f42abab5be60c079efc3673bc1611504601560',
      'accept-encoding':'gzip, deflate',
      'user-agent':'PostmanRuntime/7.1.1',
      'x-consumer-groups':'time-api, kmin',
      'x-forwarded-port':'8443',
      'x-forwarded-for':'5.186.249.11'
   },
   request_uri_args:{  

   },
   context:{  
      callbackWaitsForEmptyEventLoop:[  
         Getter/Setter
      ],
      done:[  
         Function:done
      ],
      succeed:[  
         Function:succeed
      ],
      fail:[  
         Function:fail
      ],
      logGroupName:'/aws/lambda/testKongPlugIn',
      logStreamName:'2018/03/20/[$LATEST]8255ac0135444840805de998eda7a43a',
      functionName:'testKongPlugIn',
      memoryLimitInMB:'128',
      functionVersion:'$LATEST',
      getRemainingTimeInMillis:[  
         Function:getRemainingTimeInMillis
      ],
      invokeid:'236cb636-2bd7-11e8-807e-ad685023686d',
      awsRequestId:'236cb636-2bd7-11e8-807e-ad685023686d',
      invokedFunctionArn:'arn:aws:lambda:eu-central-1:65709000000:function:testKongPlugIn'
   },
   env:{  
      PATH:'/var/lang/bin:/usr/local/bin:/usr/bin/:/bin',
      LANG:'en_US.UTF-8',
      TZ:':UTC',
      LD_LIBRARY_PATH:'/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib',
      LAMBDA_TASK_ROOT:'/var/task',
      LAMBDA_RUNTIME_DIR:'/var/runtime',
      AWS_REGION:'eu-central-1',
      AWS_DEFAULT_REGION:'eu-central-1',
      AWS_LAMBDA_LOG_GROUP_NAME:'/aws/lambda/testKongPlugIn',
      AWS_LAMBDA_LOG_STREAM_NAME:'2018/03/20/[$LATEST]8255ac0135444840805de998eda7a43a',
      AWS_LAMBDA_FUNCTION_NAME:'testKongPlugIn',
      AWS_LAMBDA_FUNCTION_MEMORY_SIZE:'128',
      AWS_LAMBDA_FUNCTION_VERSION:'$LATEST',
      _AWS_XRAY_DAEMON_ADDRESS:'169.254.79.2',
      _AWS_XRAY_DAEMON_PORT:'2000',
      AWS_XRAY_DAEMON_ADDRESS:'169.254.79.2:2000',
      AWS_XRAY_CONTEXT_MISSING:'LOG_ERROR',
      _X_AMZN_TRACE_ID:'Root=1-5ab0583c-ac1626426a7a21c00e6;Parent=45e1707d0b277232;Sampled=0',
      AWS_EXECUTION_ENV:'AWS_Lambda_nodejs6.10',
      _HANDLER:'index.handler',
      NODE_PATH:'/var/runtime:/var/task:/var/runtime/node_modules',
      AWS_ACCESS_KEY_ID:'AS<removed>LA',
      AWS_SECRET_ACCESS_KEY:'Hlhp5<removed>70+',
      AWS_SESSION_TOKEN:'<token_removed>1Jp2GiCISoUmlyGmxoii+n9Z+gPvpIht345+IciiUqcHVBQ=='
   }
}

as you can see I successfully got 'x-forwarded-for':'5.186.249.11' being that my home p for this week.

the plugin as it’s setup:

GET http://localhost:8001/apis/time/plugins/544c1760-7519-43d2-843b-74091003ab50

{
    "created_at": 1521506205328,
    "config": {
        "aws_region": "eu-central-1",
        "aws_secret": "Hlhp5<removed>70+",
        "forward_request_body": false,
        "keepalive": 60000,
        "invocation_type": "Event",
        "aws_key": "AS<removed>LA",
        "function_name": "testKongPlugIn",
        "forward_request_uri": true,
        "timeout": 60000,
        "forward_request_headers": true,
        "forward_request_method": true,
        "log_type": "Tail",
        "port": 443
    },
    "id": "544c1760-7519-43d2-843b-74091003ab50",
    "enabled": true,
    "name": "aws-lambda",
    "api_id": "62fa0ad1-14c7-47e2-8fa3-9265aa14c1bd"
}
7

Interesting…

Is your setup running behind the proxy protocol? I have a feeling this is where the issue may lie.

At some point I’ll be switching to an ALB, which doesn’t need to use the proxy protocol, so hopefully this may fix this issue going forward.

Adam

8

No,

I did not add the proxy protocol, we dont make any use not it.

Our setup is only:

Postman call > Classic Load Balancer > EC2 with Kong 12.3 (updated from this post) > AWS Lambda plugin > AWS Lambda

will update to v13 next week, will re-check if all is the same though.

9

Can you show how you have configured your:

  1. trusted_ips
  2. real_ip_header
  3. real_ip_recursive

Also can you confirm that your ELB uses proxy_protocol? Also do you have config.forward_request_headers in lambda plugin configuration?

10

Oh, I actually now know why is it like that. Core only does this:

-- X-Forwarded-* Headers
local http_x_forwarded_for = var.http_x_forwarded_for
if http_x_forwarded_for then
  var.upstream_x_forwarded_for = http_x_forwarded_for .. ", " ..
                                 realip_remote_addr

else
  var.upstream_x_forwarded_for = var.remote_addr
end

var.upstream_x_forwarded_proto = forwarded_proto
var.upstream_x_forwarded_host  = forwarded_host
var.upstream_x_forwarded_port  = forwarded_port

And these get passed to proxy module:

proxy_set_header   X-Forwarded-For   $upstream_x_forwarded_for;
proxy_set_header   X-Forwarded-Proto $upstream_x_forwarded_proto;
proxy_set_header   X-Forwarded-Host  $upstream_x_forwarded_host;
proxy_set_header   X-Forwarded-Port  $upstream_x_forwarded_port;

But lambda plugin doesn’t use proxy, if uses lua-resty-http as a HTTP client to call AWS Lambda. And now these headers don’t get injected to request. Possibly fixes:

  1. Modify lambda plugin to add these upstream headers using Nginx variables
  2. Make core set these headers to request so that lambda plugin can pick them up with config.forward_request_headers
  3. Make this part of our upcoming plugin api and the HTTP client we will have there and update lambda plugin to use that client
  4. Make lambda plugin to use proxy module instead
11

Out of those, I think 1. is the fastest route.

12

Cool! Glad you managed to find what the issue is!

Is there anything you need from me? Everything works for non-lambda stuff, so I’m pretty sure what you’ve identified is the problem :slight_smile:

13

Hi folks,

Anyone know if it is now possible to get the real client IP in x-forwarded-for header using the aws-lambda plugin? If so, how?

Thanks!