10. Kubernetes的日志与监控

Kubernetes的日志处理

• 日志对比：

传统服务的日志：

    固定机器、固定目录
    
    不受重启影响
    
    不用关注stdout、stderr

k8s服务的日志：

    节点不固定
    
    重启服务会漂移
    
    需要关注stdout、stderr

• 常见方案：

1. 远程日志     kafka、elasticsearch

2. SideCar      每个pod中运行一个sidecar，与主容器共享volume

3. LogAgent     每个node运行一个agent，daemonset方式运行

• 实践方案：

采用LogAgent方案，LogPilot + Elasticsearch + Kibana

关于LogPilot：

智能的容器日志采集工具

自动发现机制

开源 https://github.com/AliyunContainerService/log-pilot

• 主机说明：

ip	角色	cpu	内存	hostname
192.168.1.51	master	>=2	>=2G	master1
192.168.1.52	master	>=2	>=2G	master2
192.168.1.53	master	>=2	>=2G	master3
192.168.1.54	node	>=2	>=2G	node1
192.168.1.55	node	>=2	>=2G	node2
192.168.1.56	node	>=2	>=2G	node3

# kubectl get nodes

NAME    STATUS   ROLES    AGE   VERSION
node1   Ready    <none>   18h   v1.14.0
node2   Ready    <none>   18h   v1.14.0
node3   Ready    <none>   18h   v1.14.0

这里master节点不作为集群工作节点，所以没有显示，属于正常情况。

• 部署es：

# cd /software

# mkdir elk && cd elk

# vim elasticsearch.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-api
  namespace: kube-system
  labels:
    name: elasticsearch
spec:
  selector:
    app: es
  ports:
  - name: transport
    port: 9200
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-discovery
  namespace: kube-system
  labels:
    name: elasticsearch
spec:
  selector:
    app: es
  ports:
  - name: transport
    port: 9300
    protocol: TCP
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 3
  serviceName: "elasticsearch-service"
  selector:
    matchLabels:
      app: es
  template:
    metadata:
      labels:
        app: es
    spec:
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      serviceAccountName: dashboard-admin
      initContainers:
      - name: init-sysctl
        image: busybox:1.27
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        securityContext:
          privileged: true
      containers:
      - name: elasticsearch
        image: registry.cn-hangzhou.aliyuncs.com/imooc/elasticsearch:5.5.1
        ports:
        - containerPort: 9200
          protocol: TCP
        - containerPort: 9300
          protocol: TCP
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
              - SYS_RESOURCE
        resources:
          limits:
            memory: 4000Mi
          requests:
            cpu: 100m
            memory: 2000Mi
        env:                #es配置
          - name: "http.host"
            value: "0.0.0.0"
          - name: "network.host"
            value: "_eth0_"
          - name: "cluster.name"
            value: "docker-cluster"
          - name: "bootstrap.memory_lock"
            value: "false"
          - name: "discovery.zen.ping.unicast.hosts"
            value: "elasticsearch-discovery"
          - name: "discovery.zen.ping.unicast.hosts.resolve_timeout"
            value: "10s"
          - name: "discovery.zen.ping_timeout"
            value: "6s"
          - name: "discovery.zen.minimum_master_nodes"
            value: "2"
          - name: "discovery.zen.fd.ping_interval"
            value: "2s"
          - name: "discovery.zen.no_master_block"
            value: "write"
          - name: "gateway.expected_nodes"
            value: "2"
          - name: "gateway.expected_master_nodes"
            value: "1"
          - name: "transport.tcp.connect_timeout"
            value: "60s"
          - name: "ES_JAVA_OPTS"
            value: "-Xms2g -Xmx2g"
        livenessProbe:
          tcpSocket:
            port: transport
          initialDelaySeconds: 20
          periodSeconds: 10
        volumeMounts:
        - name: es-data
          mountPath: /data
      terminationGracePeriodSeconds: 30
      volumes:
      - name: es-data
        hostPath:
          path