filebeat收集应用日志

目标：收集非主进程产生的日志
背景：kubectl log只能查看主进程产生的日志。如果一些像tomcat的应用会产生单独的日志文件。
这个例子收集的是tomcat log目录下的日志。

大概结构就是这个样

环境介绍：
kubernetes：v1.11.1+icp（icp 3.1）
filebeat     6.5.0  
其他需求：过墙！ 

环境准备：

部署elasticsearch

kubernetes部署elasticsearch

部署kibana:

kubernetes部署kibana

下载镜像：docker.elastic.co/beats/filebeat-oss:6.5.0
部署的yaml文件：
filebeat.inputs.path为收集日志的路径（默认收集/mnt/log/*.log下的日志）
output.elasticsearch.hosts为es的地址

apiVersion: v1
data:
  filebeat.yml: |
    filebeat.config:
      prospectors:
        path: ${path.config}/prospectors.d/*.yml
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: false
    processors:
    - add_cloud_metadata:
    filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - /log/*.log
    output.elasticsearch:
      hosts: ['log-e-elasticsearch-client:9200']
      username: elastic
      password: changeme
kind: ConfigMap
---
kind :  Ingress
apiVersion :  extensions/v1beta1
metadata :
  name : demo-maven-tomcat
  namespace :  testcenter
spec :
  rules :
  - host : demo-maven-tomcat.testcenter.10.110.25.75.xip.io
    http :
      paths :
      - backend :
          serviceName : demo-maven-tomcat
          servicePort : 8080
        path :  /
---
kind :  Service
apiVersion :  v1
metadata :
  name :  demo-maven-tomcat
  namespace :  testcenter
  labels :
      app :  demo-maven-tomcat
spec :
  ports :
  - protocol :  TCP
    name :  http
    port : 8080
    targetPort : 8080
  selector :
      app :  demo-maven-tomcat
  sessionAffinity :  None
  type :  ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: demo-maven-tomcat
  namespace: testcenter
spec:
  podSelector:
    matchLabels:
      app: demo-maven-tomcat
  ingress:
  - ports:
    - port: 8080
      protocol: TCP
  policyTypes:
  - Ingress
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: demo-maven-tomcat
  namespace: testcenter
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: demo-maven-tomcat
    spec:
      hostNetwork: false
      initContainers:
      - name: war
        image: docker-local.repo.*.com/demomaventomcat:v23
        command: [ cp ,  /demo-maven-tomcat.war ,  /root ]
        volumeMounts:
        - mountPath: /root
          name: app-volume
        imagePullPolicy: Always
        resources:
          limits:
            cpu : 250m
            memory: 1024Mi
          requests:
            cpu : 250m
            memory: 1024Mi
      imagePullSecrets:
      - name: artifactory
      containers:
      - image: docker-local.repo.*.com/filebeat-oss:6.5.0
        name: filebeat
        volumeMounts:
        - name: app-logs
          mountPath: /log
        - name: filebeat-config
          mountPath: /etc/filebeat/
      - name: webcenter
        image: tomcat:7-jre8
        volumeMounts:
        - mountPath: /usr/local/tomcat/webapps
          name: app-volume
        - name: app-logs
          mountPath: /usr/local/tomcat/logs
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu : 250m
            memory: 1024Mi
          requests:
            cpu : 250m
            memory: 1024Mi
        livenessProbe:
          exec:
            command:
            - ls
        readinessProbe:
          exec:
            command:
            - ls
      volumes:
      - name: app-logs
        emptyDir: {}
      - name: filebeat-config
        configMap:
          name: filebeat-config
      - name: app-volume
        emptyDir: {}

整体思路为：
tomcat的log目录和filebeat收集日志的目录（path）挂载到同一个空的卷上，实现目录的共享。然后filebeat将数据发给es。