Ingress + Request transformer + Service do not work together

I have noticed this inconsistency when using an Ingress and a Request Transformer.

An Ingress specifies a path and a backend (which is a Kubernetes Service). It also defines the konghq.com/plugins annotation with one of the annotations being the request transformer plugin which rewrites/replaces the URI. As such, when this exact scenario happens, when the request is forwarded from Kong to the Kubernetes Service, the Service does not use the konghq.com/path: annotation at all!

For a more concrete example, we are using Kong to handle certain paths in the Ingress, which is forwarded to a Service of type ExternalName, which is configured with the DNS of an API Gateway.
Now, API Gateway has Stages which are basically prefixes to the URI that are added. This URI prefix is added by the Service using the konghq.com/path: /qa (in our QA environment). This works fine for all paths handled by the Ingress. However, we have certain paths which need to be rewritten to the latest paths, and for these, the /qa prefix is not prefixed when the request is forwarded from the Service to the API Gateway.

This might be related to Kong aws-lambda plugin applied to an Ingress. An unrelated note is that when things go wrong like this, it is super-hard to debug Kong internals and the documentation wasn’t great in this case. I arrived on the solution after hours of debugging. :confused:

Can someone from the Kong team (ping @hbagdi) confirm this bug or quirk in Kong?

Maybe @traines might know?

That’d be more a question for the main gateway team in the GitHub - Kong/kong: 🦍 The Cloud-Native API Gateway issues or discussions. It’s not something the Kubernetes tooling interacts with.

As written, it does look like that’s what the plugin does. kong/access.lua at 3a2d10d4a246ed629036fed6bfac1e2dd73d3d97 · Kong/kong · GitHub calls an alias of the PDK function that overrides the path entirely: kong.service.request - v3.0.x | Kong Docs