Kafka API - Add support for Proxy Protocol v1 and v2

[c4milo]

Who is this for and what problem do they have today?

When Kafka clients access Redpanda clusters from behind a NAT, Redpanda does not receive the client IPs but that of the NAT box. So configuring connection rate limits per source IP becomes unfair as a single misbehaving client will cause Redpanda to rate limit all client connections. This NAT scenario is also common in Cloud providers when people use GCP's Private Service Connect, AWS Private Links or Azure Private Links to access Redpanda clusters since all these services are implemented with NAT.

Cloud providers support the proxy protocol either in the network load balancer (AWS) or directly in the private link implementation (Azure), or they can also start up an additional service such as a target proxy in GCP. So, they will forward the information to Redpanda and Redpanda will need to learn to decode it.

What are the success criteria?

• Redpanda is able to retrieve the client's source IP when clients connect from behind NATs boxes adding proxy protocol headers.
• Redpanda correctly applies connection rate limiting per client

Why is solving this problem impactful?

Using private links in Cloud is the most secure way to expose Redpanda clusters to their users, especially when the clusters are managed by platform or shared services teams. They are more secure than VPC peerings and Transit Gateway Attachments (AWS) because the access is unidirectional, connections between two or more VPCs can only be initiated by clients. They also work across VPCs with overlapping CIDRs, unlike peerings, which is hard to avoid at scale when using IPv4. Additionally, Redpanda's ACL host filtering won't work either.

Some initial requirements

• Redpanda MUST only accept the PROXY protocol if configured to do so.
• Redpanda MUST only accept the PROXY protocol header from trusted proxies.
• Redpanda MUST not try to guess whether the PROXY protocol header is present or not.
• Every intermediary proxy may add its own header, so Redpanda should support decoding multiple headers

References

• https://seriousben.com/posts/2020-02-exploring-the-proxy-protocol/
• https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
• https://developers.cloudflare.com/spectrum/how-to/enable-proxy-protocol/
• https://learn.microsoft.com/en-us/azure/private-link/private-link-service-overview#getting-connection-information-using-tcp-proxy-v2
• https://docs.aws.amazon.com/whitepapers/latest/aws-privatelink/deploying-aws-privatelink.html
• https://techcommunity.microsoft.com/t5/fasttrack-for-azure/tcp-proxy-protocol-v2-with-azure-private-link-service-deep-dive/ba-p/3574968

JIRA Link: CORE-1296

[rockwotj]

Is this v2 support only? FWIW GCP only support v1 AFAIK: https://cloud.google.com/load-balancing/docs/tcp/setting-up-tcp#proxy-protocol

[c4milo]

ahh, good catch. Fixed! Thanks!

[mattschumpert]

FYI @dotnwat , @dlex

[emaxerrno]

https://github.com/redpanda-data/redpanda/blob/dev/src/v/rpc/transport.cc#L341

looks like expanding this in case of a config option passed and then dropping into parse() should do the trick?

though i could have sworn, i had at some point added a protocol.{h,cc} to handle generic parsing like this in the past. prolly have to go looking for it in git.

[dotnwat]

this is really interesting

[mattschumpert]

Any sense of T-shirt size @michael-redpanda ? or alternative/existing solution to unveil the source IP when connections come from behind a PrivateLink endpoint?

[michael-redpanda]

Just a few questions:

• Just for the kafka protocol interface? I think to implement this on the pp/sr/admin interface may require changes to Seastar so it can process the proxy protocol before the HTTP data
• Looking for full TLV support?

alternative/existing solution to unveil the source IP when connections come from behind a PrivateLink endpoint

I didn't see any in my cursory google search, but it's more or less an IP header with some extra bits tacked on at the end.

[michael-redpanda]

@c4milo :

Redpanda correctly applies connection rate limiting per client

Is this rate throttling (e.g. limit to X MiBps) or limit number of connections per IP?

[mattschumpert]

@michael-redpanda is this really about per-SASL-principal (#11555 )? we already have connection rate limits by IP as a cluster config, IIUC its just that the real IP is masked here.

[michael-redpanda]

@michael-redpanda is this really about per-SASL-principal (#11555 )? we already have connection rate limits by IP as a cluster config, IIUC its just that the real IP is masked here.

removed that part of the comment.

would appreciate a response to my other questions, thank you

[c4milo]

@michael-redpanda, sorry for the delay.

Just for the kafka protocol interface? I think to implement this on the pp/sr/admin interface may require changes to Seastar so it can process the proxy protocol before the HTTP data

Any service exposed by Redpanda to which customers connect to.

Is this rate throttling (e.g. limit to X MiBps) or limit number of connections per IP?

Any configuration that relies on knowing the real client IP to work correctly.