kubernetes部署Ingress-nginx

Ingress-nginx简介

Pod的IP以及service IP只能在集群内访问，如果想在集群外访问kubernetes提供的服务，可以使用nodeport、proxy、loadbalacer以及ingress等方式，由于service的IP集群外不能访问，可以使用ingress方式再代理一次，即ingress代理service，service代理pod。

Ingress基本原理图如下：

官网：https://kubernetes.github.io/ingress-nginx/
源码：https://github.com/kubernetes/ingress-nginx

下面以在阿里云自建kubernetes集群为例，使用SLB做四层代理，转发到ingress-controller节点，基本原理图如下：

备注：在私有云环境可以使用nginx四层代理配合keepalived替换上图LB。

helm部署ingress-nginx

官方参考：

https://kubernetes.github.io/ingress-nginx/deploy/#using-helm

https://github.com/kubernetes/ingress-nginx/blob/master/charts/ingress-nginx/values.yaml

集群节点：

主机名	IP地址	描述
cn-shenzhen.192.168.0.48	192.168.0.48	master节点
cn-shenzhen.192.168.0.49	192.168.0.49	ingress节点
cn-shenzhen.192.168.0.50	192.168.0.50	ingress节点
cn-shenzhen.192.168.0.51	192.168.0.51	node节点

部署方式：

• DaemonSet + nodeSeletor
• deployment设置replicas数量 + nodeSeletor + pod互斥

这里选择第一种方式，选择2个worker节点打标签，以daemonset+hostNetwork方式部署高可用ingress-nginx-controller。

添加helm chat

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm search repo -l ingress-nginx

选择两个worker节点打标签

kubectl label nodes cn-shenzhen.192.168.0.49 node=ingress
kubectl label nodes cn-shenzhen.192.168.0.50 node=ingress

部署ingress-nginx，默认镜像位于国外，需自行想办法保存到可以访问的镜像仓库，这里使用dockerhub上搜索到的镜像：

helm install nginx-ingress ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.image.registry=willdockerhub \
  --set controller.image.image=ingress-nginx-controller \
  --set controller.image.tag=v0.48.1 \
  --set controller.image.digest="" \
  --set controller.hostNetwork=true \
  --set controller.kind=DaemonSet \
  --set controller.service.type=ClusterIP \
  --set controller.hostPort.enable=true \
  --set controller.hostPort.http=80 \
  --set controller.hostPort.https=443 \
  --set controller.nodeSelector.node=ingress

查看创建的pods，被部署在标签为node=ingress的两个节点上：

[root@master ~]# kubectl -n ingress-nginx get pods -o wide
NAME                                           READY   STATUS    RESTARTS   AGE     IP             NODE                       NOMINATED NODE   READINESS GATES
nginx-ingress-ingress-nginx-controller-5wbfv   1/1     Running   0          4m11s   192.168.0.50   cn-shenzhen.192.168.0.50   <none>           <none>
nginx-ingress-ingress-nginx-controller-q9st2   1/1     Running   0          4m11s   192.168.0.49   cn-shenzhen.192.168.0.49   <none>           <none>

查看创建的service

[root@master ~]# kubectl -n ingress-nginx get svc
NAME                                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
nginx-ingress-ingress-nginx-controller             ClusterIP   172.16.94.159   <none>        80/TCP,443/TCP   4m14s
nginx-ingress-ingress-nginx-controller-admission   ClusterIP   172.16.118.78   <none>        443/TCP          4m14s

配置负载均衡

创建带有公网IP的负载均衡实例，监听四层80和443端口，转发到后端两个ingress节点hostNetwork 80和443端口：

配置域名解析，指向SLB公网IP地址：

*apps.cloudcele.com ---> 120.24.77.158

以阿里云域名为例

创建示例应用

创建两副本nginx应用、ClusterIP类型service及ingress规则：

helm repo add bitnami https://charts.bitnami.com/bitnami

helm install nginx-app bitnami/nginx \
  --namespace=apps \
  --create-namespace \
  --set replicaCount=2 \
  --set containerPorts.http=8080 \
  --set service.type=ClusterIP \
  --set service.port=80 \
  --set service.targetPortt=8080 \
  --set ingress.enabled=true \
  --set ingress.pathType=Prefix \
  --set ingress.hostname=demo.apps.cloudcele.com \
  --set ingress.path=/

查看创建的pods

[root@master ~]# kubectl -n apps get pods
NAME                         READY   STATUS    RESTARTS   AGE
nginx-app-69c694dd64-9sxsl   1/1     Running   0          13m
nginx-app-69c694dd64-rvn4d   1/1     Running   0          13m

查看创建的service

[root@master ~]# kubectl -n apps get svc
NAME        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
nginx-app   ClusterIP   172.16.215.161   <none>        80/TCP    13m

查看创建的ingress规则

[root@master ~]# kubectl -n apps get ingress
NAME        CLASS    HOSTS                     ADDRESS         PORTS   AGE
nginx-app   <none>   demo.apps.cloudcele.com   172.16.94.159   80      13m

查看ingress规则yaml配置：

[root@master ~]# kubectl -n apps get ingress nginx-app -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    meta.helm.sh/release-name: nginx-app
    meta.helm.sh/release-namespace: apps
......
spec:
  rules:
  - host: demo.apps.cloudcele.com
    http:
      paths:
      - backend:
          service:
            name: nginx-app
            port:
              name: http
        path: /
        pathType: Prefix
status:
  loadBalancer:
    ingress:
    - ip: 172.16.94.159

查看ingress运行配置：

[root@master ~]# kubectl -n apps describe ingress nginx-app
Name:             nginx-app
Namespace:        apps
Address:          172.16.94.159
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                     Path  Backends
  ----                     ----  --------
  demo.apps.cloudcele.com  
                           /   nginx-app:http (10.9.0.6:8080,10.9.0.74:8080)
Annotations:               meta.helm.sh/release-name: nginx-app
                           meta.helm.sh/release-namespace: apps
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    14m (x2 over 15m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    14m (x2 over 15m)  nginx-ingress-controller  Scheduled for sync

使用域名进行访问:

Ingress controller扩容

选择某些worker节点打标签即可：

kubectl label nodes cn-shenzhen.192.168.0.51 node=ingress

由于是daemonset类型，ingress controller pod会自动扩容到打标签的节点

[root@master ~]# kubectl -n ingress-nginx get pods -o wide
NAME                                           READY   STATUS    RESTARTS   AGE    IP             NODE                       NOMINATED NODE   READINESS GATES
nginx-ingress-ingress-nginx-controller-5wbfv   1/1     Running   0          120m   192.168.0.50   cn-shenzhen.192.168.0.50   <none>           <none>
nginx-ingress-ingress-nginx-controller-n2g7n   1/1     Running   0          54s    192.168.0.51   cn-shenzhen.192.168.0.51   <none>           <none>
nginx-ingress-ingress-nginx-controller-q9st2   1/1     Running   0          120m   192.168.0.49   cn-shenzhen.192.168.0.49   <none>           <none>

由多个独占Ingress实例组成统一接入层承载集群入口流量，同时可依据后端业务流量水平扩缩容Ingress节点。当然如果您前期的集群规模并不大，也可以采用将Ingress服务与业务应用混部的方式，但建议进行资源限制和隔离。

参考：https://developer.aliyun.com/article/575996