Kubernetes笔记：深入kubernete---日志和监控

本章中将介绍K8S使用者必须考虑的重量级问题：日志与监控。

课程中会分析当下主流的日志处理方案并选择一种方案进行日志从采集到展示的完整实践；

会讲解主流k8s监控方案prometheus，包括它的实现原理，支持的各种指标等。

常见日志采集问题和解决方案分析

传统服务VSk8s中的服务

传统服务VSk8s中的服务

 k8s的日志

 k8s的日志

k8s的日志处理方案

远端存储

远端可以选择kafka，ES，后面再做统一的处理，比如汇总到某个日志服务器

优点： 简单，使用于Docker和非Docker场景

缺点：应用需要改造，比如写入本地的应用，需要改成调用远端存储

sidecar

在每个podz中跑一个sidecar，sidecar会和主容器共享Volume，他可以访问到所有的日志文件，然后把日志文件转发到后端存储。

优点：简单，对应用没有侵入

缺点：侵入pod，消耗内存和CPU；社区不推荐

LogAgent

 在每个Node节点上部署一个Agent， 相当于sidecar方案，把sidecar从pod放在了node上。

以Daemonset方式运行，采集docker对应的json.log目录。

对于写日志文件的服务，需要服务，把容器中的日志挂载到宿主机上。同时约定好挂载的目录，LogAgent采集这个目录

优点：每个节点部署一个Agent，资源消耗小，入侵性小，对pod和应用都没有侵入

缺点：

1. 约定所有程序挂载到一个特定主机目录，文件后缀名尽量统一，否则维护困难；

2. 挂载目录是预先约定好，导致没法判断日志来自于那个pod,需要定期清理残留的日志文件。

实践方案

 

Fluentd，FileBeat：静态的，需要事先配置好。而Docker容器是动态的，容易变化。

LogPilot : 采集标准输出、错误输出，文件输出；实现静态工具的动态配置，支持Fluentd，FileBea插件

LogPilot 日志采集搭建

ES配置

---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-api
  namespace: kube-system
  labels:
    name: elasticsearch
spec:
  selector:
    app: es
  ports:
  - name: transport
    #  用于ES节点外部通信(Http)
    port: 9200
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-discovery
  namespace: kube-system
  labels:
    name: elasticsearch
spec:
  selector:
    app: es
  ports:
  - name: transport
    #  用于ES节点内部通信（TCP）
    port: 9300
    protocol: TCP
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
spec:
#  最小两个实例，保证高可用
  replicas: 3
  serviceName: "elasticsearch-service"
  selector:
    matchLabels:
      app: es
  template:
    metadata:
      labels:
        app: es
    spec:
      # 让容器可以运行在主节点上，如果通过二进制方式安装，主节点没有kubelet则是不行
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      serviceAccountName: dashboard-admin
      initContainers:
      - name: init-sysctl
        image: busybox:1.27
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        securityContext:
          privileged: true
      containers:
      - name: elasticsearch
        image: registry.cn-hangzhou.aliyuncs.com/imooc/elasticsearch:5.5.1
        ports:
        - containerPort: 9200
          protocol: TCP
        - containerPort: 9300
          protocol: TCP
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
              - SYS_RESOURCE
        resources:
          limits:
            memory: 4000Mi
          requests:
            cpu: 100m
            memory: 2000Mi
        env:
          - name: "http.host"
            value: "0.0.0.0"
          - name: "network.host"
            value: "_eth0_"
          - name: "cluster.name"
            value: "docker-cluster"
          - name: "bootstrap.memory_lock"
            value: "false"
          - name: "discovery.zen.ping.unicast.hosts"
            value: "elasticsearch-discovery"
          - name: "discovery.zen.ping.unicast.hosts.resolve_timeout"
            value: "10s"
          - name: "discovery.zen.ping_timeout"
            value: "6s"
          - name: "discovery.zen.minimum_master_nodes"
            value: "2"
          - name: "discovery.zen.fd.ping_interval"
            value: "2s"
          - name: "discovery.zen.no_master_block"
            value: "write"
          - name: "gateway.expected_nodes"
            value: "2"
          - name: "gateway.expected_master_nodes"
            value: "1"
          - name: "transport.tcp.connect_timeout"
            value: "60s"
          - name: "ES_JAVA_OPTS"
            value: "-Xms2g -Xmx2g"
        livenessProbe:
          tcpSocket:
            port: transport
          initialDelaySeconds: 20
          periodSeconds: 10
        volumeMounts:
        - name: es-data
          mountPath: /data
      terminationGracePeriodSeconds: 30
      volumes:
      - name: es-data
        hostPath:
          path: /es-data

# 创建
# kubectl apply -f elasticsearch.yaml
# 查看状态
# kubectl get svc -n kube-system
# kubectl get statefulset -n kube-system

log-pilot配置

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: log-pilot
  namespace: kube-system
  labels:
    k8s-app: log-pilot
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: log-es
        kubernetes.io/cluster-service: "true"
        version: v1.22
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      serviceAccountName: dashboard-admin
      containers:
      - name: log-pilot
        # 通过动态修改filebeat日志文件实现日志采集
        image: registry.cn-hangzhou.aliyuncs.com/imooc/log-pilot:0.9-filebeat
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        env:
          - name: "FILEBEAT_OUTPUT"
            value: "elasticsearch"
          - name: "ELASTICSEARCH_HOST"
            value: "elasticsearch-api"
          - name: "ELASTICSEARCH_PORT"
            value: "9200"
          - name: "ELASTICSEARCH_USER"
            value: "elastic"
          - name: "ELASTICSEARCH_PASSWORD"
            value: "changeme"
        volumeMounts:
        - name: sock
          # 访问宿主机docker
          mountPath: /var/run/docker.sock
        - name: root
          mountPath: /host
          readOnly: true
        - name: varlib
          mountPath: /var/lib/filebeat
        - name: varlog
          mountPath: /var/log/filebeat
        securityContext:
          capabilities:
            add:
            - SYS_ADMIN
      terminationGracePeriodSeconds: 30
      volumes:
      - name: sock
        hostPath:
          path: /var/run/docker.sock
      - name: root
        hostPath:
          path: /
      - name: varlib
        hostPath:
          path: /var/lib/filebeat
          type: DirectoryOrCreate
      - name: varlog
        hostPath:
          path: /var/log/filebeat
          type: DirectoryOrCreate

# 创建
# kubectl apply -f log-pilot.yaml
# 查看状态
# kubectl get ds -n kube-system

kibana配置

---
#Service
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: kube-system
  labels:
    component: kibana
spec:
  selector:
    component: kibana
  ports:
  - name: http
    port: 80
    targetPort: http
---
#ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana
  namespace: kube-system
spec:
  rules:
  # 记得host文件添加此域名
  - host: kibana.mooc.com
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana
          servicePort: 80
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: kibana
  namespace: kube-system
  labels:
    component: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
     component: kibana
  template:
    metadata:
      labels:
        component: kibana
    spec:
      containers:
      - name: kibana
        image: registry.cn-hangzhou.aliyuncs.com/imooc/kibana:5.5.1
        env:
        - name: CLUSTER_NAME
          value: docker-cluster
        - name: ELASTICSEARCH_URL
          value: http://elasticsearch-api:9200/
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        ports:
        - containerPort: 5601
          name: http

# 创建
# kubectl apply -f kibana.yaml
# 查看状态
# kubectl get deploy -n kube-system

应用Demo配置

#deploy
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-demo
spec:
  selector:
    matchLabels:
      app: web-demo
  replicas: 3
  template:
    metadata:
      labels:
        app: web-demo
    spec:
      containers:
      - name: web-demo
        image: hub.mooc.com/kubernetes/web:v1
        ports:
        - containerPort: 8080
        env:
        # 使用log-pilot，这个名字需要aliyun_logs_开头，如果对接ES表示索引，对接kafka表示Topic
        - name: aliyun_logs_catalina
          value: "stdout"
        - name: aliyun_logs_access
          value: "/usr/local/tomcat/logs/*"
        volumeMounts:
        - mountPath: /usr/local/tomcat/logs
          name: accesslogs
      volumes:
      # emptyDir,表示Docker自动生成的位置
      - name: accesslogs
        emptyDir: {}
---
#service
apiVersion: v1
kind: Service
metadata:
  name: web-demo
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: web-demo
  type: ClusterIP

---
#ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: web-demo
spec:
  rules:
  - host: web.mooc.com
    http:
      paths:
      - path: /
        backend:
          serviceName: web-demo
          servicePort: 80


# 创建
# kubectl apply -f web.yaml
# 查看状态
# kubectl get pods
# kubectl get pods -o wide

# 查看log-pilot日志
#docker ps | grep log-pilot
#docker logs -f 容器ID

监控

监控目的

1. 及时发现已经出现的问题
2. 提前预警可能发生的问题

监控什么

1. 系统基础指标
2. 服务基础信息
3. 服务个性化信息（QPS、处理时间等）
4. 日志（异常发生频率等）

如何监控

1. 数据采集（）
2. 数据存储（时间序列数据库）
3. 定义报警规则
4. 配置报警方式（邮件、短信等）

业内常见的监控组件/方案

1. Zabbix
2. 小米OpenFalcon
3. 听云、监控宝

Kubernetes的监控

1. 每个节点的基础指标
2. 每个容器的基础指标
3. Kubernetes集群组件

课程分享：

1. 课程分享：Docker+Kubernetes(k8s)微服务容器化实践

2. 课程分享：Kubernetes(k8s)生产级实践指南 从部署到核心应用

3. 课程分享：（极客时间）深入剖析Kubernetes