grpc在k8s中负载均衡失效事故排查

Kubernetes 的默认负载平衡通常不能与 gRPC 一起使用，在不使用 LoadBalance service 的情况下，因为 HTTP/2 链接复用特性，导致客户端的所有请求都发往同一个 Pod，导致负载不均衡。具体原因可见此文：gRPC Load Balancing on Kubernetes without Tears。原文中对 gRPC 失效的原因的摘抄如下：

However, gRPC also breaks the standard connection-level load balancing, including what’s provided by Kubernetes. This is because gRPC is built on HTTP/2, and HTTP/2 is designed to have a single long-lived TCP connection, across which all requests are multiplexed—meaning multiple requests can be active on the same connection at any point in time. Normally, this is great, as it reduces the overhead of connection management. However, it also means that (as you might imagine) connection-level balancing isn’t very useful. Once the connection is established, there’s no more balancing to be done. All requests will get pinned to a single destination pod …

问题在于 gRPC 是基于 HTTP/2 长连接的实现，gRPC 客户端与服务端建立长连接后会保持并通过这个连接持续的发送请求。而 Kubernetes Service 属于基于 TCP 连接级别的负载均衡（Connection-level Balancing），虽然支持 gRPC 通信，但其负载均衡的作用却会失效。

解决办法

客户端型负载均衡（ZooKeeper/Etcd/Consul/DNS/Nacos 等）