ingress-nginx更改默认端口/修改apiserver端口范围

问题：

直接更改ingress svc yml文件更改端口报错。

环境说明

1、 k8s部署在纯内网环境，想要外网可以访问，需要外层的服务器做nginx转发。由于公司网络限制，内网服务器和外层服务器互相通信需要使用固范围内的端口。
2、ingress默认使用的http 80；https 443；采用nodeport方式默认的端口暴露范围在 30000-32767。通过更改apiserver nodeport端口范围，达到更改ingress端口的作用。

修改apiserver端口范围

安装方式为kubeadm，会在master产生节点上会有一个文件/etc/kubernetes/manifests/kube-apiserver.yaml。添加 - --service-node-port-range=7000-8000 我设定的端口范围为7000-8000.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 172.30.210.23:5709
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.11.8
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2709
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6001
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.10.0.0/16
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --service-node-port-range=7000-8000
    image: 192.168.11.13:6010/k8s/kube-apiserver:v1.21.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.11.8
        path: /livez
        port: 5709
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 192.168.11.8
        path: /readyz
        port: 5709
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 192.168.11.8
        path: /livez
        port: 5709
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
  hostNetwork: true
  priorityClassName: system-node-critical
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
status: {}

重新启动apiserver

获取apiserver pod名：
export apiserver_pods=$(kubectl get pods --selector=component=kube-apiserver -n kube-system --output=jsonpath={.items..metadata.name})
删除上面获取的pod：
kubectl delete pod $apiserver_pods -n kube-system

更改ingress端口

1、找到部署ingress-nginx的yml文件注释如下内容：

hostNetwork: true 

不进行注释默认使用80，443端口。
更改默认端口记得注释！由于本人忘记注释，netstat -tnlp |grep 80 发现端口仍被占用，这可能就会导致ingress使用出现异常。

2、修改ingress service yml文件配置，添加nodePort后，重新部署。

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      nodePort: 6001
      targetPort: 80
      protocol: TCP
    - name: https
      port: 443
      nodePort: 6002
      targetPort: 443
      protocol: TCP
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

查看ingress-nginx使用的端口是否更改

[root@k8s-master ingress]# kubectl get svc -n gray
NAME     TYPE         CLUSTER-IP       EXTERNAL-IP    PORT(S)                     AGE
gray     nodeport    10.10.213.255     <none>         80:6001/TCP,443:6002/TCP    2m12s

ingress-nginx部署方式不同，yml文件不同，这里只是个人的解决方式，仅限参考。