多端口服务的Ingress IP-hash问题

背景

业务反馈使用Ingress的ip-hash, 同一个服务开启了http和websocket分别是两个端口, 但是配置ip-hash后, 同一个client的请求http和websocket不在同一个后端.

探究

根据业务Ingress配置,配置如下实例:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/cors-allow-origin: '*'
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 200m
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/upstream-hash-by: $binary_remote_addr
  name: hellogo
spec:
  rules:
  - host: hellogo.d.xiaomi.net
    http:
      paths:
      - backend:
          serviceName: hellogo #http1, 8080
          servicePort: 8080
        path: /8080
      - backend:
          serviceName: hellogo #http2, 9090
          servicePort: 9090
        path: /9090
      - backend:
          serviceName: hellogo #websocket, 8081
          servicePort: 8081
        path: /ws

创建多个副本

$ kubectl get po -l app=hellogo
NAME                       READY   STATUS    RESTARTS   AGE
hellogo-699f997454-b5vs4   1/1     Running   0          66m
hellogo-699f997454-hm924   1/1     Running   0          66m
hellogo-699f997454-mfbqv   1/1     Running   0          66m
hellogo-699f997454-qdrwn   1/1     Running   0          66m
hellogo-699f997454-srh9b   1/1     Running   0          66m
hellogo-699f997454-wlwfh   1/1     Running   0          66m

测试http 8080端口, 请求到pod hellogo-699f997454-qdrwn

$ curl http://hellogo.d.xiaomi.net/8080
hello 8080!
host hellogo.d.xiaomi.net
remoteaddr 10.46.23.1:15340
realip 10.232.41.102
hostname hellogo-699f997454-qdrwn 

$ curl http://hellogo.d.xiaomi.net/8080
hello 8080!
host hellogo.d.xiaomi.net
remoteaddr 10.46.23.1:15866
realip 10.232.41.102
hostname hellogo-699f997454-qdrwn

测试http 8080端口, 请求到pod hellogo-699f997454-b5vs4

$ curl http://hellogo.d.xiaomi.net/9090
hello 9090!
host hellogo.d.xiaomi.net
remoteaddr 10.38.200.195:23706
realip 10.232.41.102
hostname hellogo-699f997454-b5vs4

$ curl http://hellogo.d.xiaomi.net/9090
hello 9090!
host hellogo.d.xiaomi.net
remoteaddr 10.38.200.195:23706
realip 10.232.41.102
hostname hellogo-699f997454-b5vs4

猜想是由于获取的nginx server列表顺序不一致导致的, 但是看源码ip list是直接从endpoint获取的, 进入nginx-ingress查看

$ kubectl exec -it -n kube-system nginx-ingress-controller-m496n sh
# dbg工具查看nginx后端列表
/etc/nginx $ /dbg backends list | grep hellogo
default-hellogo-8080
default-hellogo-8081
default-hellogo-9090
# 8080端口的列表
/etc/nginx $ /dbg backends get default-hellogo-8080
{
  "endpoints": [
    {
      "address": "10.46.12.107",
      "port": "8080"
    },
    {
      "address": "10.46.12.108",
      "port": "8080"
    },
    {
      "address": "10.46.12.109",
      "port": "8080"
    },
    {
      "address": "10.46.23.23",
      "port": "8080"
    },
    {
      "address": "10.46.23.25",
      "port": "8080"
    },
    {
      "address": "10.46.23.29",
      "port": "8080"
    }
  ],
  "name": "default-hellogo-8080",
  "noServer": false,
  "port": 8080,
  ...
}
# 9090端口的列表
/etc/nginx $ /dbg backends get default-hellogo-9090
{
  "endpoints": [
    {
      "address": "10.46.12.107",
      "port": "9090"
    },
    {
      "address": "10.46.12.108",
      "port": "9090"
    },
    {
      "address": "10.46.12.109",
      "port": "9090"
    },
    {
      "address": "10.46.23.23",
      "port": "9090"
    },
    {
      "address": "10.46.23.25",
      "port": "9090"
    },
    {
      "address": "10.46.23.29",
      "port": "9090"
    }
  ],
  "name": "default-hellogo-9090",
  "noServer": false,
  "port": 9090,
  ...
}

对比发现两个端口的列表是一样的,只能看看代码.

ip-hash代码在https://github.com/kubernetes/ingress-nginx/blob/master/rootfs/etc/nginx/lua/balancer/chash.lua

function _M.new(self, backend)
  local nodes = util.get_nodes(backend.endpoints)
  local o = {
    instance = self.factory:new(nodes),  --获取后端pod ip列表
    hash_by = backend["upstreamHashByConfig"]["upstream-hash-by"],
    traffic_shaping_policy = backend.trafficShapingPolicy,
    alternative_backends = backend.alternativeBackends,
  }
  setmetatable(o, self)
  self.__index = self
  return o
end

function _M.balance(self)
  local key = util.lua_ngx_var(self.hash_by) --获取需要hash的变量
  return self.instance:find(key)  --计算hash值
end

return _M

关键是在get_nodes函数,位于https://github.com/kubernetes/ingress-nginx/blob/master/rootfs/etc/nginx/lua/util.lua

function _M.get_nodes(endpoints)
  local nodes = {}
  local weight = 1 --所有后端weight相同都为1

  for _, endpoint in pairs(endpoints) do
    local endpoint_string = endpoint.address .. ":" .. endpoint.port --endpoint为ip+port
    nodes[endpoint_string] = weight
  end

  return nodes
end

通过代码可以看到在ingress-nginx中,实际的后端(upstream)是包含端口的,通过hash计算得到的值也不一样。

解决建议

首先确认系统的架构是不是合理，不同的端口提供不同的服务，一般是相互独立的。
如果确实有类似需求：

• 通过同一个端口提供服务，使用path来区分不同功能
• 修改代码，也比较简单

---

更多文章见 Qinng’s Blog