【微服务部署】06-日志集成

1. EFK日志三件套集成

1.1 核心组件

• Elasticsearch（存储）
• Fluentd（收集器）
• Kibana（数据看板）

每个Kubernetes节点上可以运行一个Fluentd实例，这个实例会去收集所有Pod打印出来的日志流。Fluentd可以做到无侵入的收集日志；
多个Kubernetes节点组成了Kubernetes集群，每个Fluentd收集到的日志都推送到Elasticsearch中。Elasticsearch可以单独部署，也可以部署在Kuberbetes集群中；
Kibana负责管理Elasticsearch，并作为数据看板

1.2 部署

需要部署Elasticsearch、Fluentd、Kibana

Elasticsearch

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: default
  labels:
    tag: elasticsearch
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: "elasticsearch:7.5.2"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        volumeMounts:
          - mountPath: "/usr/share/elasticsearch/data"
            name: elasticsearch-storage
        ports:
        - containerPort: 9200
        - containerPort: 9300
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: elasticsearch-storage
          hostPath:
              path: "/d/k8s/volumes/elasticsearch/data"
              type: DirectoryOrCreate
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: default
  labels:
    tag: "elasticsearch"
spec:
  type: NodePort
  ports:
  - nodePort: 30007
    port: 9200
    targetPort: 9200
    name: "9200"
    protocol: TCP
  - nodePort: 30008
    port: 9300
    targetPort: 9300
    name: "9300"
    protocol: TCP
  selector:
    app: elasticsearch

Fluentd

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: fluentd
  namespace: default
  labels:
    tag: fluentd
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: "witskeeper/fluentd-es:68"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        volumeMounts:
          - mountPath: "/fluentd/etc/fluent.conf"
            name: fluentd-config
            subPath: fluent.conf
          - mountPath: "/var/log"
            name: containers-logs
          - mountPath: "/var/lib/docker/containers"
            name: docker-logs
        ports:
        - containerPort: 24224
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: fluentd-config
          configMap:
              name: fluentd-config
        - name: containers-logs
          hostPath:
              path: "/var/log"
              type: DirectoryOrCreate
        - name: docker-logs
          hostPath:
            path: /var/lib/docker/containers
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: fluentd
  namespace: default
  labels:
    tag: "fluentd"
spec:
  type: NodePort
  ports:
  - nodePort: 30011
    port: 24224
    targetPort: 24224
    protocol: TCP
  selector:
    app: fluentd

Kibana

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: kibana
  namespace: default
  labels:
    tag: kibana
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: "kibana:6.8.6"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        - name: XPACK_MONITORING_ENABLED
          value: "true"
        ports:
        - containerPort: 5601
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: default
  labels:
    tag: "kibana"
spec:
  type: NodePort
  ports:
  - nodePort: 30009
    port: 5601
    targetPort: 5601
    protocol: TCP
  selector:
    app: kibana

2. Exceptionless日志系统

2.1 Exceptionless核心特性

• 友好的界面
• 内置日志分类看板
• 按团队和项目管理应用
• 支持Web hook发送异常通知
• 基于.Net Core开源

Exceptionless由UI、Job、Api三个镜像组成。背后依赖了Redis存储和文件存储；
文件存储是用来存储应用提交的日志文件，然后通过Job将这些文件推送到Elasticsearch中，用户可以通过UI和Api从dashboard上查询异常日志；
应用程序通过Api推送异常日志

2.2 Exceptionless部署文件

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: exceptionless-api
  namespace: default
  labels:
    tag: exceptionless-api
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: exceptionless-api
    spec:
      containers:
      - name: exceptionless-api
        image: "exceptionless/api:6.0.0"
        imagePullPolicy: IfNotPresent
        env:
        - name: EX_AppMode
          value: "Production"
        - name: EX_BaseURL
          value: http://localhost:30013
        - name: EX_ConnectionStrings__Cache
          value: provider=redis
        - name: EX_ConnectionStrings__Elasticsearch
          value: server=http://elasticsearch:9200;enable-size-plugin=false
        #- name: EX_ConnectionStrings__Email
        #  value: smtps://user:password@smtp.host.com:587
        - name: EX_ConnectionStrings__MessageBus
          value: provider=redis
        - name: EX_ConnectionStrings__Queue
          value: provider=redis
        - name: EX_ConnectionStrings__Redis
          value: server=redis,abortConnect=false
        - name: EX_ConnectionStrings__Storage
          value: provider=folder;path=/app/storage
        - name: EX_RunJobsInProcess
          value: 'false'
        volumeMounts:
          - mountPath: "/app/storage"
            name: exceptionless-storage
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: exceptionless-storage
          hostPath:
              path: "/d/k8s/volumes/exceptionless" // 本地存储目录
              type: DirectoryOrCreate
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-api
  namespace: default
  labels:
    tag: "exceptionless-api"
spec:
  type: NodePort
  ports:
  - nodePort: 30012
    port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: exceptionless-api

---

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: exceptionless-job
  namespace: default
  labels:
    tag: exceptionless-job
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: exceptionless-job
    spec:
      containers:
      - name: exceptionless-job
        image: "exceptionless/job:6.0.0"
        imagePullPolicy: IfNotPresent
        env:
        - name: EX_AppMode
          value: "Production"
        - name: EX_BaseURL
          value: http://localhost:30013
        - name: EX_ConnectionStrings__Cache
          value: provider=redis
        - name: EX_ConnectionStrings__Elasticsearch
          value: server=http://elasticsearch:9200;enable-size-plugin=false
        #- name: EX_ConnectionStrings__Email
        #  value: smtps://user:password@smtp.host.com:587
        - name: EX_ConnectionStrings__MessageBus
          value: provider=redis
        - name: EX_ConnectionStrings__Queue
          value: provider=redis
        - name: EX_ConnectionStrings__Redis
          value: server=redis,abortConnect=false
        - name: EX_ConnectionStrings__Storage
          value: provider=folder;path=/app/storage
        - name: EX_RunJobsInProcess
          value: 'false'
        volumeMounts:
          - mountPath: "/app/storage"
            name: exceptionless-job-storage
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: exceptionless-job-storage
          hostPath:
              path: "/d/k8s/volumes/exceptionless"
              type: DirectoryOrCreate
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-job
  namespace: default
  labels:
    tag: "exceptionless-job"
spec:
  type: NodePort
  ports:
  - nodePort: 30014
    port: 80
    targetPort: 80
    protocol: TCP

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: exceptionless-ui
  namespace: default
  labels:
    tag: exceptionless-ui
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: exceptionless-ui
    spec:
      containers:
      - name: exceptionless-ui
        image: "exceptionless/ui:2.8.1497"
        imagePullPolicy: IfNotPresent
        env:
        - name: AppMode
          value: "Production"
        - name: EX_ApiUrl
          value: http://localhost:30012
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-ui
  namespace: default
  labels:
    tag: "exceptionless-ui"
spec:
  type: NodePort
  ports:
  - nodePort: 30013
    port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: exceptionless-ui

// Program
public static int Main(string[] args)
{
    Log.Logger = new LoggerConfiguration().ReadFrom.Configuration(Configuration)
   .MinimumLevel.Debug()
   .Enrich.FromLogContext()
   .WriteTo.Console(new RenderedCompactJsonFormatter())
   //.WriteTo.Fluentd("localhost", 30011, tag: "geektime-ordering-api", restrictedToMinimumLevel: Serilog.Events.LogEventLevel.Debug)
   .CreateLogger();
   // 将Configuration中的exceptionless读取出来,并绑定到它默认的配置上去
    Configuration.GetSection("exceptionless").Bind(Exceptionless.ExceptionlessClient.Default.Configuration);
    try
    {
        Log.Information("Starting web host");
        CreateHostBuilder(args).Build().Run();
        return 0;
    }
    catch (Exception ex)
    {
        Log.Fatal(ex, "Host terminated unexpectedly");
        return 1;
    }
    finally
    {
        Log.CloseAndFlush();
    }
}


// 配置文件
"exceptionless": {
    "ApiKey": "PSaokqmXC8T4xgWal9I0atA8TlYG7Ytz65JvxAL3",
    "ServerUrl": "http://localhost:30012"
  },

// startup
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
	app.UseExceptionless();// 放在所有中间件最前面.为了能捕获到所有异常
}

2.3 K8s中使用Exceptionless

配置文件

 "exceptionless": {
    "ApiKey": "PSaokqmXC8T4xgWal9I0atA8TlYG7Ytz65JvxAL3" // 需要是当前项目的秘钥
  },

因为serverUrl地址是共享的，地址被配置在env.txt中

要将配置引用进来,需要在deployment.yaml中注册serverUrl为环境变量

通过EFK收集应用的全量日志,Exceptionless收集异常日志