k8s创建nginx

一. yaml文件创建

1.1 创建文件

[root@master nginx-k8s]# ls
k8s-nginx.yaml  nginx-config-pvc.yaml

[root@master nginx-k8s]# vi nginx-config-pvc.yaml
# 复制 conf.d文件夹 挂载的pvc
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nginx-config-pv
spec:
  capacity:
    storage: 100Mi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Delete
  nfs:
    server: 192.168.0.27
    path: /data/k8s

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-config-pvc
  namespace: kube-system
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Mi


[root@master nginx-k8s]# vi k8s-nginx.yaml 

# 复制
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: proxy-nginx
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: proxy-nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
          protocol: TCP
        volumeMounts:
        - name: nginx-conf
          subPath: nginx-conf
          mountPath: /etc/nginx/conf.d
      volumes:
      - name: nginx-conf
        persistentVolumeClaim:
          claimName: nginx-config-pvc
      nodeSelector:
        node-role.kubernetes.io/master: ""
      tolerations:
      - key: "node-role.kubernetes.io/master"
        effect: "NoSchedule"
---
apiVersion: v1
kind: Service
metadata:
  name: proxy-nginx
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    nodePort: 32767
  selector:
    k8s-app: proxy-nginx

1.2 部署

[root@master nginx-k8s]# kubectl apply -f nginx-config-pvc.yaml 
[root@master nginx-k8s]# kubectl apply -f k8s-nginx.yaml 

1.3 配置 default.conf

进入上面 pvc挂载 nginx的conf.d的配置文件夹

[root@master nginx-k8s]# cd /data/k8s/nginx-conf/

[root@master nginx-conf]# vi default.conf 

# 复制
server {
    listen       80;
    server_name  localhost;

    #charset koi8-r;
    #access_log  /var/log/nginx/host.access.log  main;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    #location ~ \.php$ {
    #    root           html;
    #    fastcgi_pass   127.0.0.1:9000;
    #    fastcgi_index  index.php;
    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    #    include        fastcgi_params;
    #}

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}
}

1.4 重新部署

[root@master nginx-k8s]# kubectl delete -f k8s-nginx.yaml 
deployment.extensions "proxy-nginx" deleted
service "proxy-nginx" deleted
[root@master nginx-k8s]# kubectl apply -f k8s-nginx.yaml 
deployment.extensions/proxy-nginx created
service/proxy-nginx created

1.5 访问

# 查看nodePort端口
[root@master nginx-k8s]# kubectl get svc -n kube-system
NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
proxy-nginx                 NodePort    10.1.28.33     <none>        80:32767/TCP             34s

二. helm创建nginx-ingress

2.1 安装

我们这里通过Helm来简化 nginx-ingress 的安装，所以确保 Helm 能够正常使用，可以参考我们前面的文章：Kubernetes Helm 初体验。 由于 nginx-ingress 所在的节点需要能够访问外网，这样域名可以解析到这些节点上直接使用，所以需要让 nginx-ingress 绑定节点的 80 和 443 端口，所以我们这里通过 DasemonSet 和 hostPort 来进行部署，当然需要通过 nodeSelector 来筛选有外网 IP 的边缘节点。

[root@master ~]# kubectl get nodes --show-labels
NAME     STATUS   ROLES    AGE   VERSION   LABELS
master   Ready    master   18d   v1.15.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master,kubernetes.io/os=linux,node-role.kubernetes.io/master=
node1    Ready    <none>   18d   v1.15.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node1,kubernetes.io/os=linux
node2    Ready    <none>   18d   v1.15.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node2,kubernetes.io/os=linux

要获取 Chart 包的默认参数值可以通过下面的命令获取：

$ helm fetch stable/nginx-ingress
$ tar -xvf nginx-ingress-0.31.0.tgz

[root@master nginx-ingress]# ls
nginx-ingress  nginx-ingress-0.31.0.tgz

[root@master nginx-ingress]# cd nginx-ingress/

比如我们集群使用 kubeadm 安装的，而且只有 master 节点有外网 IP，所以需要将 nginx-ingress 绑定在 master 节点上，通过 nodeSelector 绑定 label 标签：kubernetes.io/hostname=master，当然还需要容忍该节点的污点，这个需要结合你的节点实际情况进行绑定，然后新建 my-values.yaml 文件来覆盖 nginx-ingress Chart 包的一些默认参数：

因为values.yaml里面有两个images要科学上网，所有修改了一下image

[root@master nginx-ingress]# cat <<EOF> my-values.yaml 
controller:
  image:
    repository: siriuszg/nginx-ingress-controller
    tag: "0.26.1"
  hostNetwork: true
  extraArgs:
    default-ssl-certificate: "kube-ops/harbor-tls"
  nodeSelector:
    kubernetes.io/hostname: master

defaultBackend:
  image:
    repository: registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64
    tag: "1.5"
  nodeSelector:
    kubernetes.io/hostname: master
EOF

default-ssl-certificate: "kube-ops/harbor-tls"：因为harbor使用了cert-manager自动生成证书，添加信任，否则docker login或者helm repo add https...都会报错x509: certificate is valid for ingress.local

nginx-ingress 目录下面的values.yaml文件即为默认参数值，大家可以根据自己的实际情况进行覆盖，然后使用下面的命令安装：

$ helm install --namespace kube-system --name nginx-ingress -f my-values.yaml .

# 镜像下载失败
helm install nginx-ingress apphub/nginx-ingress --version 1.30.3 -f my-values.yaml --namespace kube-system \
--set controller.image.repository=quay.io/kubernetes-ingress-controller/nginx-ingress-controller \
--set defaultBackend.image.repository=registry.cn-shanghai.aliyuncs.com/wanfei/defaultbackend-amd64

更新

helm upgrade --namespace kube-system nginx-ingress -f my-values.yaml .

安装完成后可以通过下面的命令查看 nginx-ingress 的 Pod 使用运行成功：

$ kubectl get pods -n kube-system | grep nginx-ingress
nginx-ingress-controller-587b4c68bf-vsqgm        1/1       Running   0          11h
nginx-ingress-default-backend-64fd9fd685-lmxhw   1/1       Running   0          1d

安装过程中需要用到gcr.io和quay.io的镜像，大家可以在 dockerhub 上搜索，pull 下来后重新打上 tag 即可。

2.2 测试

上面的 nginx-ingress 安装成功后，我们可以通过一个简单的示例来测试下：(ngdemo.yaml)

[root@master nginx-ingress]# vi ngdemo.yaml

#复制
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-nginx
spec:
  template:
    metadata:
      labels:
        app: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: my-nginx
  labels:
    app: my-nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    name: http
  selector:
    app: my-nginx
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-nginx
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: ngdemo.wanfei.com
    http:
      paths:
      - path: /
        backend:
          serviceName: my-nginx
          servicePort: 80

直接创建上面的资源对象：

$ kubectl apply -f ngdemo.yaml
deployment.extensions "my-nginx" created
service "my-nginx" created
ingress.extensions "my-nginx" created

注意我们在 Ingress 资源对象中添加了一个 annotations：kubernetes.io/ingress.class: “nginx”，这就是指定让这个 Ingress 通过 nginx-ingress 来处理。 上面资源创建成功后，然后我们可以将域名ngdemo.wanfei.com解析到nginx-ingress所在的边缘节点中的任意一个，当然也可以在本地/etc/hosts中添加对应的隐射也可以，然后就可以通过域名进行访问了。

到这里就证明nginx-ingress安装成功了，除此之外，我们还可以利用cert-manager来进行 HTTPS 自动化，可以参考前面的文章：Kubernetes Ingress 自动化 HTTPS，nginx-ingress还有非常多的高级配置功能，大家可以直接查看文档：https://kubernetes.github.io/ingress-nginx/。

2.3 解决上传文件大小问题

k8s nginx-ingress 上传文件大小限制