kuberbetes的服务暴露Ingress及负载均衡

时间记录：2019-7-24
我们在使用k8s的时候发现我们部署的应用需要通过服务的方式来将服务保留到外网，常用的方式有NodePort,LoadBalancer,Ingress之前使用的是NodePort的方式来进行服务的暴露的(探索阶段)。但是这种方式在将所有的节点上都打开一个端口，然后所有的流量的都从这个端口到对应的服务上，然后对应的服务再请求分配到挂载到应用上的某一个（分配的依据未研究）。这样就将端口和服务绑定起来了，且端口的使用个数是有限制的，尝试使用Ingress的方式暴露服务。
尝试的是基于ngnix的 lngress Controller，部署的基本过程分为以下几步。

基于的版本为ingress-nginx-nginx-0.25.0

创建对应的用户[RBAC]
由于lngress Controller需要访问api server在配置的时候需要制定用户，不然在启动的日志里会发现访问被禁。

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount //这个在后面会使用
  namespace: default

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: default
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
      - create
      - update

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: default

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: default

2：配置访问错误服务
由于访问的错误需要定一个服务，这里是必须要的，不然会在lngress Controller也会报错，提示未指定错误服务
这里的yaml在: 【ingress-nginx-nginx-0.25.0\ingress-nginx-nginx-0.25.0\docs\examples\customization\custom-errors】

apiVersion: v1
kind: Service
metadata:
  name: nginx-errors
  namespace: default
  labels:
    app.kubernetes.io/name: nginx-errors
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    app.kubernetes.io/name: nginx-errors
    app.kubernetes.io/part-of: ingress-nginx
  ports:
  - port: 80
    targetPort: 8080
    name: http

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-errors
  namespace: default
  labels:
    app.kubernetes.io/name: nginx-errors
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: nginx-errors
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: nginx-errors
        app.kubernetes.io/part-of: ingress-nginx
    spec:
      containers:
      - name: nginx-error-server
        image: quay.io/kubernetes-ingress-controller/custom-error-pages-amd64:0.3
        ports:
        - containerPort: 8080
        # Setting the environment variable DEBUG we can see the headers sent 
        # by the ingress controller to the backend in the client response.
        # env:
        # - name: DEBUG
        #   value: "true"

3：配置 lngress Controller ( Nginx + Ingres Controller)
部署lngress Controller，这里其实是分为Nginx + Ingres Controller两个部分，但是同一叫 lngress Controller ，nginx大家不陌生做为代理使用，lngress Controller主要是和api server 交互，然将对应的配置写到nginx的config中。
这里的yaml在【ingress-nginx-nginx-0.25.0\ingress-nginx-nginx-0.25.0\docs\examples\static-ip】

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: default
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
    spec:
      # hostNetwork makes it possible to use ipv6 and to preserve the source IP correctly regardless of docker configuration
      # however, it is not a hard dependency of the nginx-ingress-controller itself and it may cause issues if port 10254 already is taken on the host
      # that said, since hostPort is broken on CNI (https://github.com/kubernetes/kubernetes/issues/31307) we have to use hostNetwork where CNI is used
      # like with kubeadm
      hostNetwork: true #这里默认是注释的，要打开注释不然监听不到宿主机的端口
      serviceAccountName: nginx-ingress-serviceaccount #指定用户，不指定用户启动会报访问api server的禁止错误
      terminationGracePeriodSeconds: 60
      containers:
      - image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.18.0
        name: nginx-ingress-controller
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/nginx-ingress-lb
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend #指定错误的服务，用户访问错误的路径

注意：这里官网的yaml文件中需要修改的地方，不然启动会报错的
【+】hostNetwork注释打开，不然监听不到宿主机的端口
【+】指定用户名，不然访问api server 会出错
【+】指定错误的访问的服务，不然会报错提示没有指定
【+】全部在同一namespace

4：配置ingress controller 的服务

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: default
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

5：配置 Ingress
配置对应的服务发现，不然无法访问，这里用了一个现成的简单网页作为测试例子

apiVersion: extensions/v1beta1      
kind: Ingress       
metadata:           
  name: ingress-myapp   
  namespace: default     
spec:     
  rules:   
  #- host: myapp.magedu.com   这里不建议写上，如果写上了需要修改host,为dns解析而用，不写就可以为你的实际部署的节点的ip进行访问
    http:
      paths:       
      - path: / #为你的访问的路径      
        backend:    
          serviceName: htmlcpp # 你的服务的名称，确保服务下的sector标签和你的pod一致
          servicePort: 80  # 确保为你的服务的端口

以上就是配置完成的内容，我们就可以找到ingress controller实际的节点我这里为192.168.141.131，然后加上对应的访问路径，我这里就为根路径。
https://192.168.141.131/

总结：现阶段只了解了基本的使用，还有nginx的配置优化在这里如何使用等需继续学习

参考文章：
k8s&ingress

git资源位置：
ingress-nginx

时间记录：2019-7-24