Http Log Performance

Questions
1

Hello! I am using Kong as an API gateway in front of some restful API’s. As I am doing some load testing, I have found that the Http Log plugin is fairly slow. For example - If I make 1000 requests through the Kong API to my backend and they are processed around 50 TPS (So approx 20 seconds total time) - the Kong Http Log plugin works - however it takes upwards of 15 minutes for the plugin to process and send the 1000 messages to my http server (Simple Node.JS/Express app). I have verified it is not an issue with the Node application - looking at the load balancer logs in front of my NodeJS application - we are getting less than 1 TPS from the Http Log Plugin. I spent time looking into the code of the Http Log Plugin - it looks like it is using a queuing mechanism to send the logs async which is great, however I need much higher throughput. I also looked into the TCP log plugin - it looks like it fires off the messages right away (no queuing). Should I move to the TCP log plugin?

Any help is greatly appreciated!

2

Care to post the JSON settings you ran with for the plugin config? How many Nginx worker processes you run with? I use a variant of the HTTP Log plugin as well and like the functionality of a batch queue then send design but I can’t speak much to noticing a hindrance on throughput @ 50 TPS.

3

I am using the default values for nginx.

Here is the JSON config:

{
“created_at”: 1557758803,
“config”: {
“flush_timeout”: 2,
“http_endpoint”: “http://host.docker.internal:6000/gateway/audit”,
“retry_count”: 10,
“timeout”: 100,
“queue_size”: 1,
“keepalive”: 60000,
“content_type”: “application/json”,
“method”: “POST”
},
“id”: “62f5fb8c-8f40-4de4-aeb5-6bd17c4bb873”,
“service”: null,
“name”: “http-log”,
“protocols”: [“http”, “https”],
“enabled”: true,
“run_on”: “first”,
“consumer”: null,
“route”: null,
“tags”: null
}

4

bump. Looking for recommendations here!

5

bump - any help here is appreciated

6

I won’t be able to really give much assistance towards your local setup. But these are the relevant configs we run in prod via the http log plugin:

 "config": {
    "timeout": 10000,
    "method": "POST",
    "keepalive": 60000,
    "flush_timeout": 30,
    "queue_size": 20,
    "retry_count": 5
  },

And we generally achieve logging at a 150+ TPS rate currently without any 15 minute delay presenting itself.

Edit - My other advise would be to fork the HTTP plugin or drop in overwritten files within that plugin with print statements, find out if there is any weird tcp behavior going on there between src and desto and it supports keepalive, and also just to log and get a judge on how quickly its firing with some print statements from Kong side.

1 Like
7

thank you very much for a response - i will take a look at our config and retest.