【kubernetes 系列5】Kubernetes服务发现 ingress & ingress controller

Kubernetes服务发现 ingress & ingress controller

1、介绍

kubernetes Ingess 是有2部分组成，Ingress Controller 和Ingress服务组成，常用的Ingress Controller 是ingress-nginx.
工作的原理是： Ingress Controller会动态感知集群中的Ingress的规则变化，然后读取，动态生成Nginx的配置文件，最后注入到运行nginx的pod的中，然后会自动reload，配置生效。

用kubernetes Ingress 是由于它是7层调度，可以直接卸载https会话，代理的后端的pod可以直接使用明文的http协议。而Service NodePort得类型，是4层得调度，做不到这点，然而现在https是一种趋势，所以在kubernetes 对外暴露服务得时候我们通常会选择Ingress。
(ps：不过对于熟悉nginx配置，不想花时间去了解ingress-nginx的朋友，推荐直接使用nginx。 ingress-nginx定义了很多新的规则，还是需要花蛮多时间的去学习的，而且没有什么不能nginx不能完成的亮点)

2、Ingress Nginx部署

Ingress Nginx部署请参照官方的文档（https://kubernetes.github.io/ingress-nginx/）。 以下是我在实际应用场景下的配置,只需要下载run一下就可以成功部署

2.1 配置文件下载和安装

链接：https://pan.baidu.com/s/1sDHQLc6UAK9MIwm5HiE_Qg 
提取码：zada

通过下面的命令安装。

kubectl apply -f .

2.2 配置详解

ingress-controller.yaml    ：ingress-controller控制器，即ingress pod
ingress-common.yaml            ：ingress 一些公用配置，定义一些全局配置
ingress-apps.yaml                : ingress 的路由规则，其实就是nginx.conf。配置所有的业务逻辑
service-nodeport.yaml     ：暴露服务的service服务，一般直接对接在LBS的后端；
demon.yaml                            ：非必要， 测试的pod
custom-headers.yaml            ：非必要，proxy-set-headers的参数配置

2.2.1  ingress控制器配置

 

# ingress-controller.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ns-ukr
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      containers:
        - name: nginx-ingress-controller
          image: willdockerhub/nginx-ingress-controller:0.21.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=kubernetes.ukr
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1

注意点：
# 换成自己的域名空间，这里是ns-ukr。我的应用全部都在这个域名空间下
namespace: ns-ukr
# 修改掉默认的annotations的前缀。在ingress-apps中使用的annotation配置全部都是我这个自定义的前缀
- --annotations-prefix=kubernetes.ukr
# 国内可以直接下载的镜像源
image: willdockerhub/nginx-ingress-controller:0.21.0 

2.2.2  ingress 的基础配置

这里主要配置了基本配置文件，账号，角色，绑定授权。

# ingress-common.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ns-ukr
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
#  proxy-set-headers: "ns-ukr/custom-headers"
  proxy-body-size: "100m"
  ssl-redirect: "false"
  
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ns-ukr
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ns-ukr
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ns-ukr
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ns-ukr
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ns-ukr
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ns-ukr

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ns-ukr

注意点：
# 这里配置为全局配置，定义了一些nginx参数。 需要注意的这里的参数配置和nginx配置不完全相同，需要按照ingress-nginx的规则进行配置（特反感这点，为什么不支持nginx的统一配置，重新定义呢？）
data:
  proxy-set-headers: "ns-ukr/custom-headers"
  proxy-body-size: "100m"
  ssl-redirect: "false"
# namespace 全部都设置为ns-ukr
namespace: ns-ukr

2.2.3   ingress 转发规则
这里主要定义了业务的转发规则，充当nginx功能。

# ingress-nginx.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-apps
  namespace: ns-ukr
  annotations:
    kubernetes.ukr/ingress.class: "nginx"
    kubernetes.ukr/ssl-redirect: "false"
    kubernetes.ukr/enable-access-log: "true"
    kubernetes.ukr/enable-rewrite-log: "true"
spec:
  tls:
  - hosts:
    - admin.shfuchen.net
    secretName: admin-ingress-secret
  - hosts:
    - mini.shfuchen.net
    secretName: mini-ingress-secret
  - hosts:
    - game.shfuchen.net
    secretName: game-ingress-secret
  - hosts:
    - store.shfuchen.net
    secretName: store-ingress-secret
  rules:
  - host: mini.shfuchen.net
    http:
      paths:
      - path: /h5/
        backend:
          serviceName: nginx
          servicePort: 80
      - path: /static/
        backend:
          serviceName: nginx
          servicePort: 80
      - path: /favicon.ico
        backend:
          serviceName: nginx
          servicePort: 80
      - path: /
        backend:
          serviceName: app-api
          servicePort: 8080
  - host: game.shfuchen.net
    http:
      paths:
      - path:
        backend:
          serviceName: app-game
          servicePort: 8080          
  - host: www.shfuchen.net
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  - host: admin.shfuchen.net
    http:
      paths:
      - path:
        backend:
          serviceName: app-admin
          servicePort: 8080
  - host: store.shfuchen.net
    http:
      paths:
      - path:
        backend:
          serviceName: app-store
          servicePort: 8080

注意点：
# 1.可以简单的使用ingress-nginx提供的annotation来配置。（同样看上去简单，实则你要先了解每个注解组件的功能说明）
annotations:
  kubernetes.ukr/ingress.class: "nginx"
  kubernetes.ukr/ssl-redirect: "false"
  kubernetes.ukr/enable-access-log: "true"
  kubernetes.ukr/enable-rewrite-log: "true"

# 2.namespace 全部都设置为ns-ukr
namespace: ns-ukr

# 3.tls配置 如果请求是https的话，需要配置证书。
spec:
  tls:
  - hosts:
    - admin.shfuchen.net
    secretName: admin-ingress-secret
  - hosts:
    - mini.shfuchen.net
    secretName: mini-ingress-secret

# 4.多路径配置
  rules:
  - host: mini.shfuchen.net
    http:
      paths:
      - path: /h5/
        backend:
          serviceName: nginx
          servicePort: 80
      - path: /static/
        backend:
          serviceName: nginx
          servicePort: 80
      - path: /
        backend:
          serviceName: app-api
          servicePort: 8080

# 5.多域名配置
  - host: game.shfuchen.net
    http:
      paths:
      - path:
        backend:
          serviceName: app-game
          servicePort: 8080          
  - host: www.shfuchen.net
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80

2.2.3   证书配置
一般的实验实例中，都是生成证书，网上实例很多这里不做讲解。 实际应用中都可以向域名提供商申请免费的证书（阿里云，腾讯云，百度云都有）。 直接安装即可

# 安装证书

kubectl create secret tls admin-ingress-secret --cert=admin.crt --key=admin.key -n ns-ukr

这里一定需要带用namespace， 不同namespace下的证书，ingress是无法使用的。

2.2.4   服务暴露
ingress-nginx 理解上完全可以看成一个nginx服务器（实际也就是一个nginx pod）， 所以这个需要暴露出端口，供外网访问。一般都会用于SLB的服务节点。 

# service-node.yaml 

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ns-ukr
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
      nodePort: 30081
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
      nodePort: 30082
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

在这里我同时暴露出http和https两个端口，接下来就是尽情的对接吧。