云计算之kubernetes服务暴露——ingress 排错

上一回留下一个问题，就是使用浏览器无法访问到，这一回继续排错。
1. 排错思路整理
    1) nginx-ingress-controller
        pod nginx-ingress-controller-6d96ccd6f4-g2d6b , master与node 都可以访问
        kubectl describe pod nginx-ingress-controller-6d96ccd6f4-g2d6b -n ingress-nginx
            Name:         nginx-ingress-controller-6d96ccd6f4-g2d6b
            Namespace:    ingress-nginx
            Node:         192.168.89.133/192.168.89.133
            Start Time:   Wed, 08 Jan 2020 09:34:34 +0800
            Labels:       app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          pod-template-hash=6d96ccd6f4
            Annotations:  kubernetes.io/limit-ranger: LimitRanger plugin set: cpu, memory request for container nginx-ingress-controller
                          prometheus.io/port: 10254
                          prometheus.io/scrape: true
            Status:       Running
            IP:           10.1.20.3 # 这个为nginx-ingress-controller pod 的ip
            IPs:
              IP:           10.1.20.3
            Controlled By:  ReplicaSet/nginx-ingress-controller-6d96ccd6f4
            Containers:
              nginx-ingress-controller:
                Container ID:  docker://20d0722863e8896d830b43f730c8b233009e0a81c9ab1841f805a761ca5d235e
                Image:         192.168.89.132:5000/nginx-ingress-controller:0.26.2
                Image ID:      docker-pullable://192.168.89.132:5000/nginx-ingress-controller@sha256:417cf5edc00c1581a44f67c1b1713b55db3811834498f45fc5de40fbe5f6a8de
                Ports:         80/TCP, 443/TCP
                Host Ports:    0/TCP, 0/TCP
                Args:
                  /nginx-ingress-controller
                  --configmap=$(POD_NAMESPACE)/nginx-configuration
                  --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
                  --udp-services-configmap=$(POD_NAMESPACE)/udp-services
                  --publish-service=$(POD_NAMESPACE)/ingress-nginx
                  --annotations-prefix=nginx.ingress.kubernetes.io
                State:          Running
                  Started:      Wed, 08 Jan 2020 14:27:53 +0800
                Last State:     Terminated
                  Reason:       Error
                  Exit Code:    137
                  Started:      Wed, 08 Jan 2020 09:35:24 +0800
                  Finished:     Wed, 08 Jan 2020 14:27:40 +0800
                Ready:          True
                Restart Count:  1
                Requests:
                  cpu:      100m
                  memory:   90Mi
                Liveness:   http-get http://:10254/healthz delay=10s timeout=10s period=10s #success=1 #failure=3 , 健康性检测 curl http://10.1.20.3:10254/healthz
                Readiness:  http-get http://:10254/healthz delay=0s timeout=10s period=10s #success=1 #failure=3
                Environment:
                  POD_NAME:       nginx-ingress-controller-6d96ccd6f4-g2d6b (v1:metadata.name)
                  POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
                Mounts:
                  /var/run/secrets/kubernetes.io/serviceaccount from nginx-ingress-serviceaccount-token-mz87t (ro)
            Conditions:
              Type              Status
              Initialized       True
              Ready             True
              ContainersReady   True
              PodScheduled      True
            Volumes:
              nginx-ingress-serviceaccount-token-mz87t:
                Type:        Secret (a volume populated by a Secret)
                SecretName:  nginx-ingress-serviceaccount-token-mz87t
                Optional:    false
            QoS Class:       Burstable
            Node-Selectors:  kubernetes.io/os=linux
            Tolerations:     <none>
            Events:          <none>
        curl http://10.1.20.3:10254/healthz
            ok
            过了一会儿只有89.133这一台可以访问，奇怪! 说明控制器在正常工作，没有问题，不过只有一台可以正常访问接口，不知道这是不是正常的。
        kubectl get svc -n ingress-nginx -o wide
            NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE    SELECTOR
            ingress-nginx   NodePort   169.169.178.15   <none>        80:30080/TCP,443:30443/TCP   164m   app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
            service类型为NodePort , service ip : 169.169.178.15 , 该ip两台node可以ping通，master无法ping通 ， 该地址为nginx-ingress-controller pod对应的service
            kubectl describe svc ingress-nginx -n ingress-nginx
                Name:                     ingress-nginx
                Namespace:                ingress-nginx
                Labels:                   app.kubernetes.io/name=ingress-nginx
                                          app.kubernetes.io/part-of=ingress-nginx
                Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/par...
                Selector:                 app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
                Type:                     NodePort
                IP:                       169.169.178.15
                Port:                     http  80/TCP
                TargetPort:               80/TCP
                NodePort:                 http  30080/TCP
                Endpoints:                10.1.20.3:80
                Port:                     https  443/TCP
                TargetPort:               443/TCP
                NodePort:                 https  30443/TCP
                Endpoints:                10.1.20.3:443
                Session Affinity:         None
                External Traffic Policy:  Cluster
                Events:                   <none>
            curl http://169.169.178.15
                <html>
                <head><title>404 Not Found</title></head>
                <body>
                <center><h1>404 Not Found</h1></center>
                <hr><center>openresty/1.15.8.2</center>
                </body>
                </html>
            curl http://169.169.178.15/healthz 回应为空 , 除了 /healthz ， 访问其他都报404
    2). ingress.yaml 规则
        less ingress.yaml
            spec:
              # 路由规则
              rules:
              # 主机名，只能是域名，修改为你自己的
              - host: k8s.aaa.com
                http:
                  paths:
                  - path:
                    backend:
                      # 后台部署的 Service Name，与上面部署的 Tomcat 对应
                      serviceName: tomcat-http
                      # 后台部署的 Service Port，与上面部署的 Tomcat 对应
                      servicePort: 8080
            配置文件最下面部分可知，访问 k8s.aaa.com ，跳转到service名为tomcat-http，端口为8080的服务，思考：这个k8s.aaa.com映射和谁一起？
        kubectl get Ingress
            NAME        HOSTS         ADDRESS          PORTS   AGE
            nginx-web   k8s.aaa.com   169.169.178.15   80      21h
            ADDRESS 169.169.178.15为nginx-ingress的服务地址 ， 由此可知，Ingress.yaml没有创建容器，而只是创建了一种路由规则，给这个路由规则起了一个名字: nginx-web
            k8s.aaa.com -> 169.169.178.15 -> 10.1.20.3 -> 起了一个nginx容器，upstream定义了调度规则 -> tomcat-http -> 169.169.70.125(tomcat-http服务ip) -> tomcat-app[ 10.1.27.4(192.168.89.134)/10.1.20.5(192.168.89.133) ] -> 访问到tomcat服务
        为了不引起干扰，把master和两台node /etc/hosts里有关k8s.aaa.com的映射都删了。
        k8snode01: ip a
            6: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default
                link/ether fe:fb:9b:d5:31:d0 brd ff:ff:ff:ff:ff:ff
                inet 169.169.0.1/32 brd 169.169.0.1 scope global kube-ipvs0
                   valid_lft forever preferred_lft forever
                inet 169.169.0.10/32 brd 169.169.0.10 scope global kube-ipvs0
                   valid_lft forever preferred_lft forever
                inet 169.169.70.125/32 brd 169.169.70.125 scope global kube-ipvs0
                   valid_lft forever preferred_lft forever
                inet 169.169.178.15/32 brd 169.169.178.15 scope global kube-ipvs0
                   valid_lft forever preferred_lft forever
            发现这个ip 169.169.178.15 在192.168.89.133这台机器上被解析在kube-ipvs0上了。
        思考： 访问k8s.aaa.com ， 就是访问169.169.178.15 , 就是访问192.168.89.133 ?
        找一台机器：dnsserver(192.168.89.128):
            /etc/hosts:
                192.168.89.133  k8s.aaa.com
            curl k8s.aaa.com
                curl: (7) Failed connect to k8s.aaa.com:80; Connection refused
            不行，这样解析不行，咋弄？
        k8snode01:
            curl -D- http://169.169.178.15  -H 'Host: k8s.aaa.com'
            可以出来
        将192.168.89.133  k8s.aaa.com 写到windows hosts 里；
        访问加上30080端口，
        http://k8s.aaa.com:30080/
        终于出来了，访问的时候是要加端口的
    3). 为了验证一下是不是这个问题，在测试一个服务
        vim test-ingress-pod.yaml
            apiVersion: v1
            kind: Service
            metadata:
              name: myapp-svc
              namespace: default
            spec:
              selector:
                app: myapp
                env: test
              ports:
              - name: http
                port: 80
                targetPort: 80
            ---
            apiVersion: apps/v1
            kind: Deployment
            metadata:
              name: myapp-test
            spec:
              replicas: 2
              selector:
                matchLabels:
                  app: myapp
                  env: test
              template:
                metadata:
                  labels:
                    app: myapp
                    env: test
                spec:
                  containers:
                  - name: myapp
                    image: nginx:1.15-alpine
                    ports:
                    - name: httpd
                      containerPort: 80
        vim test-ingress-myapp.yaml
            apiVersion: extensions/v1beta1
            kind: Ingress
            metadata:
              name: ingress-myapp
              namespace: default
              annotations:
                kubernetes.io/ingress.class: "nginx"
            spec:
              rules:
              - host: www.tchua.top
                http:
                  paths:
                  - path: /
                    backend:
                      serviceName: myapp-svc
                      servicePort: 80
        kubectl create -f test-ingress-pods.yaml
        kubectl create -f test-ingress-myapp.yaml
        windows 里配置 192.168.89.133 www.tchua.top
        浏览器访问： http://www.tchua.top:30080 , nginx页面出来了，问题解决
2. 总结
    有些问题出现了，一路排错排不出来，可能参考一下其他人的文档，灵感思路就来了。此处要点：这里访问时需要加上svc映射到主机时产生的nodePort端口号，service-nodeport.yaml中定义
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
      nodePort: 30080
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
      nodePort: 30443
    有时候就是一个小小的问题，就能卡好几天，一定不要急躁，休息一下，忙点其他东西，再一看，就出来了，另外，对应用的原理还是要特别熟悉，再整理下这个ingress流程：
        ingress是为了给集群外部暴露服务使用的，首先要部署好ingress controller 控制器，我们选择的是nginx ingress controller，官网上有对应的搭建流程，nginx-ingress-controller镜像比较大，直接构建比较耗时，可以提前下载好导入本地仓库。控制器弄好后，创建测试应用，pod,services等部署好后，再部署ingress.yaml，这个文件编写的是调度规则，给nginx controller控制器使用，要查看ingress controller控制器在哪一台，以及这个服务映射到节点上的端口，进行解析，浏览器访问的时候需要加上端口号。
        因为上面为测试用例，如果在真实环境，可以将新的域名解析到公司内部dns服务器上，访问的时候还要使用端口号，这个问题也好解决，在其前部加一个nginx调度，使用标准端口调度到后面的指定的解析端口上即可。
        本地应用案例流程：
            a. 将tchua.top域名解析到公司内部dns服务器上
                dnsserver:
                    vim /etc/named.conf
                        ......
                        zone "tchua.top" IN {
                            type master;
                            file "tchua.top.zone";
                        };
                        ......
                    cp -a /var/named/web.com.zone /var/named/tchua.top.zone
                    vim /var/named/tchua.top.zone
                        $TTL 1D
                        @  IN SOA @ tchua.top. (
                                            0  ; serial
                                            1D ; refresh
                                            1H ; retry
                                            1W ; expire
                                            3H )   ; minimum
                            NS @
                            A  192.168.89.133
                        www       A  192.168.89.133
                    systemctl restart named
                    hosts文件即可去掉，直接浏览器访问 http://www.tchua.top:30080
            b. 去掉端口30080访问
                修改映射端口http://:30080到常用端口80
                vim service-nodeport.yaml
                    ...
                    - name: http
                      port: 80
                      targetPort: 80
                      protocol: TCP
                      nodePort: 80
                    ...
                kubectl apply -f service-nodeport.yaml
                kubectl get svc -n ingress-nginx
                    NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                   AGE
                    ingress-nginx   NodePort   169.169.178.15   <none>        80:80/TCP,443:30443/TCP   5h7m
                    发现端口已修改
                浏览器访问： http://k8s.aaa.com  和  http://www.tchua.top 都可以访问到，验证通过，可以正常使用