K8S笔记0605

Ingress 

K8s集群对外暴露服务的方式目前只有三种：
Loadblancer

Nodeport

ingress

Ingress是kubernetes APl中的标准资源类型之一,ingress实现的功能是在应用层对客户端请求的host名称或请求的URL路径把请求转发到指定的service资源的规则，即用于将kubernetes集群外部的请求资源转发之集群内部的service，再被service转发之pod处理客户端的请求。

Ingress资源需要指定监听地址、请求的host和URL等配置，然后根据这些规则的匹配机制将客户端的请求进行转发，这种能够为ingress配置资源监听并转发流量的组件称为ingress控制器(ingress controller)，ingress controller是kubernetes的一个附件，类似于dashboard或者flannel 一样，需要单独部署。

ingress组成：

1. Ingress Controller：负载均衡实现
2. ingress服务：将Nginx的配置抽象成一个Ingress对象，每添加一个新的服务只需写一个新的Ingress的yaml文件即可

ingress里有三个必要字段：

• host：访问该应用的域名，也就是域名解析

• serverName：应用的service名称

• serverPort：service端口

ingress工作流程
 

 客户端首先对 ngdemo.qikqiak.com 执行 DNS 解析，得到 Ingress Controller 所在节点的 IP，然后客户端向 Ingress Controller 发送 HTTP 请求，然后根据 Ingress 对象里面的描述匹配域名，找到对应的 Service 对象，并获取关联的 Endpoints 列表，将客户端的请求转发给其中一个 Pod。

案例一

安装

  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx-admission
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: ingress-nginx-controller-admission
      namespace: ingress-nginx
      path: /networking/v1/ingresses
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: validate.nginx.ingress.kubernetes.io
  rules:
  - apiGroups:
    - networking.k8s.io
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - ingresses
  sideEffects: None

案例二 

每个节点部署ingress-nginx-controller并且使用host(宿主机网络)形式来提高性能

[root@k8s-master1 1.1-ingress-nginx-1.2.0_deploy-yaml]# cat 2.ingress-nginx-controller-v1.2.0_daemonset.yaml 
apiVersion: v1
kind: Namespace
metadata:
  labels:
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  name: ingress-nginx
---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx
  namespace: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx-admission
  namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx
  namespace: ingress-nginx
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - configmaps
  - pods
  - secrets
  - endpoints
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses/status
  verbs:
  - update
- apiGroups:
  - networking.k8s.io
  resources:
  - ingressclasses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resourceNames:
  - ingress-controller-leader
  resources:
  - configmaps
  verbs:
  - get
  - update
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx-admission
  namespace: ingress-nginx
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - nodes
  - pods
  - secrets
  - namespaces
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses/status
  verbs:
  - update
- apiGroups:
  - networking.k8s.io
  resources:
  - ingressclasses
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx-admission
rules:
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - validatingwebhookconfigurations
  verbs:
  - get
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx
subjects:
- kind: ServiceAccount
  name: ingress-nginx
  namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx-admission
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
  name: ingress-nginx-admission
  namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx
subjects:
- kind: ServiceAccount
  name: ingress-nginx
  namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx-admission
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
  name: ingress-nginx-admission
  namespace: ingress-nginx
---
apiVersion: v1
data:
  allow-snippet-annotations: "true"
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx-controller
  namespace: ingress-nginx
---
#apiVersion: v1
#kind: Service
#metadata:
#  labels:
#    app.kubernetes.io/component: controller
#    app.kubernetes.io/instance: ingress-nginx
#    app.kubernetes.io/name: ingress-nginx
#    app.kubernetes.io/part-of: ingress-nginx
#    app.kubernetes.io/version: 1.2.0
#  name: ingress-nginx-controller
#  namespace: ingress-nginx
#spec:
#  ipFamilies:
#  - IPv4
#  ipFamilyPolicy: SingleStack
#  ports:
#  - appProtocol: http
#    name: http
#    port: 80
#    protocol: TCP
#    targetPort: http
#  - appProtocol: https
#    name: https
#    port: 443
#    protocol: TCP
#    targetPort: https
#  selector:
#    app.kubernetes.io/component: controller
#    app.kubernetes.io/instance: ingress-nginx
#    app.kubernetes.io/name: ingress-nginx
#  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx-controller-admission
  namespace: ingress-nginx
spec:
  ports:
  - appProtocol: https
    name: https-webhook
    port: 443
    targetPort: webhook
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: ClusterIP
---
apiVersion: apps/v1
#kind: Deployment
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  minReadySeconds: 0
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
    spec:
      hostNetwork: true #使用宿主机网络
      hostPID: true #使用宿主机Pid
      containers:
      - args:
        - /nginx-ingress-controller
        - --election-id=ingress-controller-leader
        - --controller-class=k8s.io/ingress-nginx
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: LD_PRELOAD
          value: /usr/local/lib/libmimalloc.so
        image: registry.cn-hangzhou.aliyuncs.com/zhangshijie/ingress-nginx:v1.2.0 
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - /wait-shutdown
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        - containerPort: 8443
          name: webhook
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 100m
            memory: 90Mi
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL
          runAsUser: 101
        volumeMounts:
        - mountPath: /usr/local/certificates/
          name: webhook-cert
          readOnly: true
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
      - name: webhook-cert
        secret:
          secretName: ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx-admission-create
  namespace: ingress-nginx
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.2.0
      name: ingress-nginx-admission-create
    spec:
      containers:
      - args:
        - create
        - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
        - --namespace=$(POD_NAMESPACE)
        - --secret-name=ingress-nginx-admission
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        image: registry.cn-hangzhou.aliyuncs.com/zhangshijie/kube-webhook-certgen:v1.1.1 
        imagePullPolicy: IfNotPresent
        name: create
        securityContext:
          allowPrivilegeEscalation: false
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: OnFailure
      securityContext:
        fsGroup: 2000
        runAsNonRoot: true
        runAsUser: 2000
      serviceAccountName: ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx-admission-patch
  namespace: ingress-nginx
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.2.0
      name: ingress-nginx-admission-patch
    spec:
      containers:
      - args:
        - patch
        - --webhook-name=ingress-nginx-admission
        - --namespace=$(POD_NAMESPACE)
        - --patch-mutating=false
        - --secret-name=ingress-nginx-admission
        - --patch-failure-policy=Fail
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        image: registry.cn-hangzhou.aliyuncs.com/zhangshijie/kube-webhook-certgen:v1.1.1 
        imagePullPolicy: IfNotPresent
        name: patch
        securityContext:
          allowPrivilegeEscalation: false
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: OnFailure
      securityContext:
        fsGroup: 2000
        runAsNonRoot: true
        runAsUser: 2000
      serviceAccountName: ingress-nginx-admission
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: nginx
spec:
  controller: k8s.io/ingress-nginx
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx-admission
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: ingress-nginx-controller-admission
      namespace: ingress-nginx
      path: /networking/v1/ingresses
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: validate.nginx.ingress.kubernetes.io
  rules:
  - apiGroups:
    - networking.k8s.io
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - ingresses
  sideEffects: None

 案例二-一 

[root@k8s-master1 Ingress-case]# cat 2.1.ingress_single-host.yaml 
#apiVersion: networking.k8s.io/v1beta1
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-web
  namespace: magedu
  annotations:
    kubernetes.io/ingress.class: "nginx" ##指定Ingress Controller的类型
    nginx.ingress.kubernetes.io/use-regex: "true" ##指定后面rules定义的path可以使用正则表达式
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600" ##连接超时时间,默认为5s
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600" ##后端服务器回转数据超时时间,默认为60s
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600" ##后端服务器响应超时时间,默认为60s
    nginx.ingress.kubernetes.io/proxy-body-size: "50m" ##客户端上传文件，最大大小，默认为20m
    #nginx.ingress.kubernetes.io/rewrite-target: / ##URL重写
    nginx.ingress.kubernetes.io/app-root: /index.html

#spec:
#  rules: #路由规则
#  - host: www.jiege.com ##客户端访问的host域名
#    http:
#      paths:
#      - path:
#        backend:
#          serviceName: magedu-nginx-service  #转发至哪个service
#          servicePort: 80 ##转发至service的端口号

spec:
  rules:
  - host: www.jiege.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: magedu-tomcat-app1-service 
            port:
              number: 80

[root@k8s-master1 Ingress-case]# cat tomcat-app1.yaml 
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:
  labels:
    app: magedu-tomcat-app1-deployment-label
  name: magedu-tomcat-app1-deployment
  namespace: magedu
spec:
  replicas: 1
  selector:
    matchLabels:
      app: magedu-tomcat-app1-selector
  template:
    metadata:
      labels:
        app: magedu-tomcat-app1-selector
    spec:
      containers:
      - name: magedu-tomcat-app1-container
        image: k8s-harbor.com/project/tomcat-app1:2022-06-01_21_45_38
        #command: ["/apps/tomcat/bin/run_tomcat.sh"]
        #imagePullPolicy: IfNotPresent
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          protocol: TCP
          name: http
        env:
        - name: "password"
          value: "123456"
        - name: "age"
          value: "18"
        resources:
          limits:
            cpu: 1
            memory: "512Mi"
          requests:
            cpu: 500m
            memory: "512Mi"
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: magedu-tomcat-app1-service-label
  name: magedu-tomcat-app1-service
  namespace: magedu
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
    nodePort: 40003
  selector:
    app: magedu-tomcat-app1-selector

 案例二-二 存在多个url 按照url转发

[root@k8s-master1 Ingress-case]# cat 2.2.ingress_multi-host.yaml 
#apiVersion: networking.k8s.io/v1beta1
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-web
  namespace: magedu
  annotations:
    kubernetes.io/ingress.class: "nginx" ##指定Ingress Controller的类型
    nginx.ingress.kubernetes.io/use-regex: "true" ##指定后面rules定义的path可以使用正则表达式
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600" ##连接超时时间,默认为5s
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600" ##后端服务器回转数据超时时间,默认为60s
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600" ##后端服务器响应超时时间,默认为60s
    nginx.ingress.kubernetes.io/proxy-body-size: "10m" ##客户端上传文件，最大大小，默认为20m
    #nginx.ingress.kubernetes.io/rewrite-target: / ##URL重写
    nginx.ingress.kubernetes.io/app-root: /index.html

spec:
  rules:
  - host: www.jiege.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: magedu-tomcat-app1-service
            port:
              number: 80


  - host: mobile.jiege.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: magedu-tomcat-app2-service
            port:
              number: 80

 案例二- 三 一个域名多个url

[root@k8s-master1 Ingress-case]# cat 3.1.ingress-url.yaml 
#apiVersion: networking.k8s.io/v1beta1
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-web
  namespace: magedu
  annotations:
    kubernetes.io/ingress.class: "nginx" ##指定Ingress Controller的类型
    nginx.ingress.kubernetes.io/use-regex: "true" ##指定后面rules定义的path可以使用正则表达式
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600" ##连接超时时间,默认为5s
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600" ##后端服务器回转数据超时时间,默认为60s
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600" ##后端服务器响应超时时间,默认为60s
    nginx.ingress.kubernetes.io/proxy-body-size: "10m" ##客户端上传文件，最大大小，默认为20m
    #nginx.ingress.kubernetes.io/rewrite-target: / ##URL重写
    nginx.ingress.kubernetes.io/app-root: /index.html
spec:
  rules:

  - host: www.jiege.com
    http:
      paths:
      - pathType: Prefix
        path: "/app1"
        backend:
          service:
            name: magedu-tomcat-app1-service
            port:
              number: 80

      - pathType: Prefix
        path: "/app2"
        backend:
          service:
            name: magedu-tomcat-app2-service
            port:
              number: 80

 https进行访问

从外部到ingress是https加密，ingress到http是明文访问

自签发证书

openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 3560 -nodes -subj '/CN=www.jiegeca.com'

 openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=www.jiege.com'

openssl x509 -req -sha256 -days 3650 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt

 

 证书上传至k8s

kubectl  create secret tls  tls-secret-www(证书名称)     --cert=server.crt     --key=server.key      -n magedu #第一种方法通过文件直接创建
kubectl  create secret tls  tls-secret-mobile  --cert=server2022.crt --key=server2022.key  -n magedu #等于第一个方法，
kubectl  create secret    generic tls-secret --from-file=tls.crt=server.crt --from-file=tls.key=server.key -n magedu #第二种方法

[root@k8s-master1 Ingress-case]# cat 4.1.ingress-https-magedu_single-host.yaml 
#apiVersion: networking.k8s.io/v1beta1
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-web
  namespace: magedu
  annotations:
    kubernetes.io/ingress.class: "nginx" ##指定Ingress Controller的类型
    nginx.ingress.kubernetes.io/ssl-redirect: 'true' #SSL重定向，即将http请求强制重定向至https，等于nginx中的全站https
spec:
  tls:
  - hosts:
    - www.jiege.com
    secretName: tls-secret-www #命令kubectl get secrets -n magedu

  rules:
  - host: www.jiege.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: magedu-tomcat-app1-service
            port:
              number: 80

 多域名https证书

root@k8s-master1 Ingress-case]# cat 4.2.ingress-https-magedu_multi-host.yaml 
#apiVersion: networking.k8s.io/v1beta1
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-web-mobile
  namespace: magedu
  annotations:
    kubernetes.io/ingress.class: "nginx" ##指定Ingress Controller的类型
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'
spec:
  tls:
  - hosts:
    - mobile.jiege.com
    secretName: tls-secret-mobile 
  - hosts:
    - www.jiege.com
    secretName: tls-secret-www
  rules:
  - host: www.jiege.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: magedu-tomcat-app1-service
            port:
              number: 80


  - host: mobile.jiege.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: magedu-tomcat-app2-service
            port:
              number: 80

openssl req -new -newkey rsa:4096 -keyout mobile.key -out mobile.csr -nodes -subj '/CN=mobile.jiege.com

openssl x509 -req -sha256 -days 3650 -in mobile.csr -CA ../certs/ca.crt -CAkey ../certs/ca.key -set_serial 01  -out mobile.crt
kubectl  create secret tls  tls-secret-mobile     --cert=mobile.crt     --key=mobile.key  -n magedu 

 

curl -lvk https://mobile.jiege.com/myapp/index.html #排错

更新证书

 加密server.crt，server.key文件

将加密后的进行修改

kubectl edit secrets tls-secret-www -n magedu

 

 Prometheus

一 监控简介

通过业务监控系统，全面掌握业务环境的运行状态，通过白盒监控能够提前预知业务瓶颈，通过黑盒监控能够第一时间发现业务故障并通过告警通告运维人员进行紧急恢复，从而将业务影响降到最低。

黑盒监控，关注的是时时的状态，一般都是正在发生的事件，比如nginx web界面打开的是界面报错503、磁盘无法报错数据等，即黑盒监控重点在于能对正在发生的故障进行通知告警。

白盒监控，关注的是原因，也就是系统内部暴露的一些指标数据，比如nginx后端服务器的响应时长、磁盘的工/o负载值等。

监控系统需要能够有效的支持白盒监控和黑盒监控，通过白盒能够了解其内部的实际运行状态，以及对监控指标的观察能够预判可能出现的潜在问题，从而对潜在的不确定因素进行提前优化并避免问题的发生，而通过黑盒监控，比如常见的如HTTP探针、TCP探针等，可以在系统或者服务在发生故障时能够快速通知相关的人员进行处理，通过建立完善的监控体系，从而达到以下目的: 

长期趋势分析:通过对监控样本数据的持续收集和统计，对监控指标进行长期趋势分析。例如，通过对磁盘空间增长率的判断，我们可以提前预测在未来什么时间节点上需要对资源进行扩容。

对照分析:两个版本的系统运行资源使用情况的差异如何?在不同容量情况下系统的并发和负载变化如何?通过监控能够方便的对系统进行跟踪和比较。

告警:当系统出现或者即将出现故障时，监控系统需要迅速反应并通知管理员，从而能够对问题进行快速的处理或者提前预防问题的发生，避免出现对业务的影响。

故障分析与定位︰当问题发生后，需要对问题进行调查和处理。通过对不同监控监控以及历史数据的分析，能够找到并解决根源问题。

数据可视化:通过可视化仪表盘能够直接获取系统的运行状态、资源使用情况、以及服务运行状态等直观的信息。

常见的监控方案

开源监控软件: cacti、nagios、zabbix、 smokeping、open-falcon等

 Prometheus简介

 Prometheushttps://github.com/prometheus是一个开源系统监控和警报工具包

Prometheus的主要特点是

• 具有由指标名称和键/值对标识的时间序列数据的多维数据模型
• PromQL，一种灵活的查询语言，可利用这种维度
• 不依赖分布式存储;单个服务器节点是自治的
• 时序收集通过 HTTP 上的拉取模型进行
• 通过中间网关支持推送时间序列
• 通过服务发现或静态配置发现目标
• 多种模式的绘图和仪表板支持

 Prometheus监控部署

Prometheus架构

prometheus server︰主服务，接受外部http请求，收集、存储与查询数据等(9090端口) 

prometheus targets :静态收集的目标服务数据(主动拉取数据)

service discovery︰动态发现服务，可以发现虚拟机和pod内的服务

prometheus alerting:报警通知，对事件进行判断并把告警信息推送到Alertmanager

push gateway︰数据收集代理服务器(类似于zabbix proxy )

data visualization and export:数据可视化与数据导出(访问客户端)

 部署方式

https://prometheus.io/download/#官方二进制下载及安装,prometheus server的监听端口为9090
https://prometheus.io/docs/prometheus/latest/installation/#docker镜像直接启动
https: //github.com/coreos/kube-prometheus #operator部署
~# apt install prometheus #使用apt或者yum安装

 1.1、解压二进制程序

ln -sv /apps/prometheus-2.36.1.linux-amd64 /apps/prometheus

 使用测试工具检测

用于检测prometheus配置文件、检测metrics数据等

[root@lvs-backup prometheus]# ./promtool check config prometheus.yml 
Checking prometheus.yml
 SUCCESS: prometheus.yml is valid prometheus config file syntax

1.2、创建service文件启动脚本

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network.target


[Service]
Restart=on-failure
WorkingDirectory=/apps/prometheus/
ExecStart=/apps/prometheus/prometheus --config.file=/apps/prometheus/prometheus.yml --web.enable-lifecycle（配置热加载执行命令curl -X POST http://192.168.226.152:9090/-/reload）


[Install]
WantedBy=multi-user.target

1.3、启动prometheus

systemctl daemon-reload && systemctl restart prometheus && systemctl enable prometheus

1.4、验证web界面

 2.1、二进制安装node-exporter

k8s各node节点使用二进制或者daemonset方式安装node_exporter，用于收集各k8s node节点宿主机的监控指标数据，默认监听端口为9100。

vim /etc/systemd/system/node-exporter.service
[Unit]
Description=Prometheus Server
After=network.target

[Service]
ExecStart=/apps/node_exporter/node_exporter

[Install]
WantedBy=multi-user.target


systemctl daemon-reload && systemctl restart node-exporter.service && systemctl enable node-exporter.service

 添加节点

vim prometheus/prometheus.yml
 - job_name: "prometheus-node"
    static_configs:
      - targets: ["localhost:9090","192.168.226.145:9100","192.168.226.152:9100","192.168.226.146:9100"]

curl -X POST http://192.168.226.152:9090/-/reload

 

 Grafana部署

grafana是一个可视化组件，用于接收客户端浏览器的请求并连接到prometheus查询数据，最后经过渲染并在浏览器进行体系化显示，需要注意的是，grafana查询数据类似于zabbix一样需要自定义模板，模板可以手动制作也可以导入已有模板。

下载官方包并安装 

https://grafana.com/grafana/download/8.5.4?edition=oss#官方网站
wget https://dl.grafana.com/oss/release/grafana-8.5.4-1.x86_64.rpm

yum install -y urw-fonts    #安装依赖

rpm -i grafana-8.5.4-1.x86_64.rpm    #安装


sudo /bin/systemctl daemon-reload && sudo /bin/systemctl enable grafana-server.service && systemctl start grafana-server.service    #设置开机自启

验证web界面 

初始账号密码都是admin

 添加prometheus数据源

 

grafana 导入模板

https://grafana.com/grafana/dashboards/

 

 

 

 

 prometheus绘图

PromQL数据基础

数据分类

• 瞬时向量、瞬时数据(instant vector):是一组时间序列，每个时间序列包含单个数据样本，比如node_memory_MemFree_bytes查询当前剩余内存就是一个瞬时向量，该表达式的返回值中只会包含该时间序列中的最新的一个样本值，而相应的这样的表达式称之为瞬时向量表达式，例如:例如:prometheus_http_requests_total
• 范围向量、范围数据(range vector) :是指在任何一个时间范围内，抓取的所有度量指标数据.比如最近一天的网卡流量趋势图，例如:prometheus_http_requests_total [ 5m ]
• 标量、纯量数据( scalar):是一个浮点数类型的数据值，使用node_loadl获取到时一个瞬时向量，但是可用使用内置函数scalar( )将瞬时向量转换为标量，例如:scalar [ sum ( node_load1 ))
• 字符串( string):简单的字符串类型的数据,目前未使用 (a simple string value; currently unused) .

数据类型

• Counter:计数器,Counter类型代表一个累积的指标数据，在没有被重置的前提下只增不减，比如磁盘IO总数、nginx的请求总数、网卡流经的报文总数等。
• Gauge:仪表盘,Gauge类型代表一个可以任意变化的指标数据，值可以随时增高或减少，如带宽速录、CPU负载、内存利用率、nginx活动连接数等。
• Histogram:累积直方图，Histogram会在一段时间范围内对数据进行采样(通常是请求持续时间或响应大小等)，假如每分钟产生一个当前的活跃连接数，那么一天就会产生1440个数据，查看数据的每间隔的绘图跨度为2小时，那么2点的柱状图( bucket)会包含o点到2点即两个小时的数据，而4点的柱状图( bucket)则会包含o点到4点的数据，而6点的柱状图( bucket)则会包含0点到6点的数据。
• Summary:摘要，也是一组数据，统计的不是区间的个数而是统计分位数,从o到1，表示是0%~1008，如下统计的是 0、0.25、0.5、0.75、1的数据量分别是多少

指标数据

node_memory_MemTotal_bytes #查询node节点总内存大小

node_memory_MemFree_bytes #查询node节点剩余可用内存

node_memory_MemTotal_bytes{instance="172.31.7.111:9100"}#基于标签查询指定节点的总内存

node_memory_MemFree_bytes{instance="172.31.7.111:9100"}#基于标签查询指定节点的可用内存

node_disk_io_time_seconds_total{device="sda" }#查询指定磁盘的每秒磁盘io

node_filesystem_free_bytes{device=" /dev/sdal" ,fstype="xfs " ,mountpoint=" /"}#查看指定磁盘的磁盘剩余空间

 匹配器

=r选择与提供的字符串完全相同的标签，精确匹配。

!=:选择与提供的字符串不相同的标签，去反。

=~:选择正则表达式与提供的字符串(或子字符串）相匹配的标签。

!~:选择正则表达式与提供的字符串(或子字符串)不匹配的标签。

 

 时间范围

s -秒

m-分钟

h -小时

d -天

w-周

y -年

#瞬时向量表达式，选择当前最新的数据node_memory_MemTotal_bytes{} 

#区间向量表达式，选择以当前时间为基准，查询所有节点node_memory_MemTotal_bytes

指标5分钟内的数据

node_memory_MemTotal_bytes{} [ 5m]

#区间向量表达式，选择以当前时间为基准，查询指定节点node_memory_MemTotal_bytes指标5分钟内的数据

node_memory_MemTotal_bytes{instance="192.168.226.144:9100", job="prometheus-node"}[1m]
 

 运算符

+ 加

- 减

* 乘

/ 除

% 模

^ 幂

node_memory_MemTotal_bytes{instance="192.168.226.144:9100", job="prometheus-node"}/1024/1024
#将内存进行单位从字节转行为兆

聚合运算 

max、min、avg

计算每个节点的最大的流量值

max ( node_network_receive_bytes_total) by (instance)

 sum、count

sum ( prometheus_http_requests_total) #最近总共请求数为xxx次，用于计算返回值的总数(如http请求次数) 

count ( node_os_version )2#可以用于统计节点数、pod数量等

count_values ( )#对value的个数(行数)进行计数

count_values( "node_version " ,node_os_version)#统计不同的系统版本节点有多少

abs、abssent 内置函数

 abs ( )#返回指标数据的值 abs (sum( prometheus_http_requests_total{handler=" /metrics" } ) )

absent ( ) #如果监指标有数据就返回空，如果监控项没有数据就返回1，可用于对监控项设置告警通知absent ( sum( prometheus_http_requests_total { handler= "/metrics" }) )

stddev、stdvar

 stddev ( )#标准差 stddev(prometheus_http_requests_total)#5+5=10,1+9=10,1+9这一组的数据差异就大，在系统是数据波动较大，不稳定

stdvar ( )#求方差 stdvar( prometheus_http_requests_total)

topk、bottomk

topk ( ) #样本值排名最大的N个数据 

#取从大到小的前6个 topk(6, prometheus_http_requests_total)

bottomk ( )#样本值排名最小的N个数据

#取从小到大的前6个 bottomk ( 6, prometheus_http_requests_total)

rate、irate

 rate()#函数是专门搭配counter数据类型使用函数，功能是取counter数据类型在这个时间段中平均每秒的增量平均数,适合用于计算数据相对平稳的数据。

rate(prometheus_http_requests_total [ 5m ] )rate ( node_network_receive_bytes_total [ 5m ] )

irate( )#函数是专门搭配counter数据类型使用函数, irate获取的是指定时间范围内最近的两个数据来计算数据的速率，适合计算数据变化比较大的数据，显示的数据相对比较准确。

irate( prometheus_http_requests_total [ 5m ])

irate ( node_network _receive_bytes_total [ 5m ] )

by、without

#by，在计算结果中，只保留by指定的标签的值，并移除其它所有的

sum(rate(node_network_receive_packets_total {instance=~”.*") [ 10m] )) by(instance)

sum(rate( node_memory_MemFree_bytes [ 5m] ) ) by (increase)

#without，从计算结果中移除列举的instance ,job标签，保留其它标签sum ( prometheus_http_requests_total) without ( instance,job)