kubernetes上部署ELK集群

最新推荐文章于 2024-07-11 16:07:32 发布

Dustin.Hoffman

最新推荐文章于 2024-07-11 16:07:32 发布

阅读量5.3k

点赞数 5

本文链接：https://blog.csdn.net/weixin_43334786/article/details/122435701

版权

kubernetes 同时被 3 个专栏收录

18 篇文章 7 订阅

订阅专栏

elk

16 篇文章 3 订阅

订阅专栏

kafka

3 篇文章 0 订阅

订阅专栏

一、系统架构

IP	角色	主机名	pod
192.168.1.3	master	master1	es-master,es-data,es-client,zookeeper,kafka,filebeat
192.168.1.4	master	master2	es-master,es-data,es-client,zookeeper,kafka,filebeat
192.168.1.5	master	master3	es-master,es-data,es-client,zookeeper,kafka,filebeat
192.168.1.6	node	node1	es-data,filebeat
192.168.1.7	node	node2	es-data,kibana,filebeat,logstash

二、创建StorageClass和Namespace

1、创建StorageClass

[root@master1 ~]# mkdir -p /elk/{master,data,client,kibana,zookeeper,kafka}
[root@master1 ~]# mkdir elk && cd elk
[root@master1 elk]# cat 00-sc.yaml 
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

2、创建Namespace

[root@master1 elk]# cat 00-ns.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: ns-elk
  labels:
    app: elasticsearch

3、创建证书

参考：https://github.com/elastic/helm-charts/blob/master/elasticsearch/examples/security/Makefile#L24-L35

[root@master1 elk]# docker run --name es-certutil -i -w /tmp elasticsearch:7.16.2 /bin/sh -c  \
    "elasticsearch-certutil ca --out /tmp/es-ca.p12 --pass '' && \
    elasticsearch-certutil cert --name security-master --dns \
    security-master --ca /tmp/es-ca.p12 --pass '' --ca-pass '' --out /tmp/elastic-certificates.p12"
[root@master1 elk]# docker cp es-certutil:/tmp/elastic-certificates.p12 ./

生成 ssl 认证要使用的 secret elastic-certificates：

[root@master1 elk]# kubectl -n ns-elk create secret generic elastic-certificates --from-file=./elastic-certificates.p12

三、部署ES集群

1、Master节点

（1）创建3个master节点的PV

[root@master1 elk]# cat 00-pv-master.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-pv-0
  namespace: ns-elk
  labels:
    name: local-storage-pv-0
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /elk/master
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master1
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-pv-1
  namespace: ns-elk
  labels:
    name: local-storage-pv-1
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /elk/master
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master2
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-pv-2
  namespace: ns-elk
  labels:
    name: local-storage-pv-2
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /elk/master
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master3

（2）部署master节点

[root@master1 elk]# cat 01-es-master.yaml 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: ns-elk
  name: elasticsearch-master
  labels:
    app: elasticsearch
    role: master
spec:
  serviceName: elasticsearch-master
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
      role: master
  template:
    metadata:
      labels:
        app: elasticsearch
        role: master
    spec:
      containers:
        - name: elasticsearch
          image: elasticsearch:7.16.2
          command: ["bash", "-c", "ulimit -l unlimited && sysctl -w vm.max_map_count=262144 && chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data && exec su elasticsearch docker-entrypoint.sh"]
          ports:
            - containerPort: 9200
              name: http
            - containerPort: 9300
              name: transport
          env:
            #- name: discovery.seed_hosts
            #  value: "elasticsearch-master-0.elasticsearch-master,elasticsearch-master-1.elasticsearch-master,elasticsearch-master-2.elasticsearch-master"
            - name: discovery.seed_hosts
              value: "elasticsearch-master-0.elasticsearch-master,elasticsearch-master-1.elasticsearch-master,elasticsearch-master-2.elasticsearch-master,elasticsearch-data-0.elasticsearch-data,elasticsearch-data-1.elasticsearch-data,elasticsearch-data-2.elasticsearch-data,elasticsearch-data-3.elasticsearch-data,elasticsearch-data-4.elasticsearch-data,elasticsearch-client-0.elasticsearch-client,elasticsearch-client-1.elasticsearch-client,elasticsearch-client-2.elasticsearch-client"
            - name: cluster.initial_master_nodes
              value: "elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2"
            - name: ES_JAVA_OPTS
              value: -Xms512m -Xmx512m
            - name: node.master
              value: "true"
            - name: node.ingest
              value: "false"
            - name: node.data
              value: "false"
            - name: cluster.name
              value: "elasticsearch"
            - name: node.name
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: xpack.security.enabled
              value: "true"
            - name: xpack.security.transport.ssl.enabled
              value: "true"
            - name: xpack.monitoring.collection.enabled
              value: "true"
            - name: xpack.security.transport.ssl.verification_mode
              value: "certificate"
            - name: xpack.security.transport.ssl.keystore.path
              value: "/usr/share/elasticsearch/config/elastic-certificates.p12"
            - name: xpack.security.transport.ssl.truststore.path
              value: "/usr/share/elasticsearch/config/elastic-certificates.p12"

          volumeMounts:
           - mountPath: /usr/share/elasticsearch/data
             name: pv-storage-elastic-master
           - name: elastic-certificates
             readOnly: true
             mountPath: "/usr/share/elasticsearch/config/elastic-certificates.p12"
             subPath: elastic-certificates.p12
           - mountPath: /etc/localtime
             name: localtime
          securityContext:
            privileged: true
      volumes:
      - name: elastic-certificates
        secret:
          secretName: elastic-certificates
      - hostPath:
          path: /etc/localtime
        name: localtime

  volumeClaimTemplates:
  - metadata:
      name: pv-storage-elastic-master
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-storage"
      resources:
        requests:
          storage: 1Gi

2、Data节点

（1）创建data节点的PV

[root@master1 elk]# cat 00-pv-data.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-data-0
  namespace: elasticsearch
  labels:
    name: local-storage-data-0
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /elk/data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master1
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-data-1
  namespace: elasticsearch
  labels:
    name: local-storage-data-1
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /elk/data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master2
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-data-2
  namespace: elasticsearch
  labels:
    name: local-storage-data-2
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /elk/data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master3
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-data-3
  namespace: elasticsearch
  labels:
    name: local-storage-data-3
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /elk/data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-data-4
  namespace: elasticsearch
  labels:
    name: local-storage-data-4
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /elk/data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node2

（2）部署Data节点

[root@master1 elk]# cat 02-es-data.yaml 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: ns-elk
  name: elasticsearch-data
  labels:
    app: elasticsearch
    role: data
spec:
  serviceName: elasticsearch-data
  replicas: 5
  selector:
    matchLabels:
      app: elasticsearch
      role: data
  template:
    metadata:
      labels:
        app: elasticsearch
        role: data
    spec:
      containers:
        - name: elasticsearch
          image: elasticsearch:7.16.2
          command: ["bash", "-c", "ulimit -l unlimited && sysctl -w vm.max_map_count=262144 && chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data && exec su elasticsearch docker-entrypoint.sh"]
          ports:
            - containerPort: 9200
              name: http
            - containerPort: 9300
              name: transport
          env:
            #- name: discovery.seed_hosts
            #  value: "elasticsearch-data-0.elasticsearch-data,elasticsearch-data-1.elasticsearch-data,elasticsearch-data-2.elasticsearch-data,elasticsearch-data-3.elasticsearch-data,elasticsearch-data-4.elasticsearch-data"
            - name: discovery.seed_hosts
              value: "elasticsearch-master-0.elasticsearch-master,elasticsearch-master-1.elasticsearch-master,elasticsearch-master-2.elasticsearch-master,elasticsearch-data-0.elasticsearch-data,elasticsearch-data-1.elasticsearch-data,elasticsearch-data-2.elasticsearch-data,elasticsearch-data-3.elasticsearch-data,elasticsearch-data-4.elasticsearch-data,elasticsearch-client-0.elasticsearch-client,elasticsearch-client-1.elasticsearch-client,elasticsearch-client-2.elasticsearch-client"
            - name: cluster.initial_master_nodes
              value: "elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2"
            - name: ES_JAVA_OPTS
              value: -Xms512m -Xmx512m
            - name: node.master
              value: "false"
            - name: node.ingest
              value: "false"
            - name: node.data
              value: "true"
            - name: cluster.name
              value: "elasticsearch"
            - name: node.name
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: xpack.security.enabled
              value: "true"
            - name: xpack.security.transport.ssl.enabled
              value: "true"
            - name: xpack.monitoring.collection.enabled
              value: "true"
            - name: xpack.security.transport.ssl.verification_mode
              value: "certificate"
            - name: xpack.security.transport.ssl.keystore.path
              value: "/usr/share/elasticsearch/config/elastic-certificates.p12"
            - name: xpack.security.transport.ssl.truststore.path
              value: "/usr/share/elasticsearch/config/elastic-certificates.p12"

          volumeMounts:
           - mountPath: /usr/share/elasticsearch/data
             name: pv-storage-elastic-data
           - name: elastic-certificates
             readOnly: true
             mountPath: "/usr/share/elasticsearch/config/elastic-certificates.p12"
             subPath: elastic-certificates.p12
           - mountPath: /etc/localtime
             name: localtime

          securityContext:
            privileged: true

      volumes:
      - name: elastic-certificates
        secret:
          secretName: elastic-certificates
      - hostPath:
          path: /etc/localtime
        name: localtime

  volumeClaimTemplates:
  - metadata:
      name: pv-storage-elastic-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-storage"
      resources:
        requests:
          storage: 2Gi

3、部署Client节点

（1）创建client节点的PV

apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: ns-elk
  name: elasticsearch-client
  labels:
    app: elasticsearch
    role: client
spec:
  serviceName: elasticsearch-client
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
      role: client
  template:
    metadata:
      labels:
        app: elasticsearch
        role: client
    spec:
      containers:
        - name: elasticsearch
          image: elasticsearch:7.16.2
          command: ["bash", "-c", "ulimit -l unlimited && sysctl -w vm.max_map_count=262144 && chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data && exec su elasticsearch docker-entrypoint.sh"]
          ports:
            - containerPort: 9200
              name: http
            - containerPort: 9300
              name: transport
          env:
            #- name: discovery.seed_hosts
            #  value: "elasticsearch-client-0.elasticsearch-client,elasticsearch-client-1.elasticsearch-client,elasticsearch-client-2.elasticsearch-client"
            - name: discovery.seed_hosts
              value: "elasticsearch-master-0.elasticsearch-master,elasticsearch-master-1.elasticsearch-master,elasticsearch-master-2.elasticsearch-master,elasticsearch-data-0.elasticsearch-data,elasticsearch-data-1.elasticsearch-data,elasticsearch-data-2.elasticsearch-data,elasticsearch-data-3.elasticsearch-data,elasticsearch-data-4.elasticsearch-data,elasticsearch-client-0.elasticsearch-client,elasticsearch-client-1.elasticsearch-client,elasticsearch-client-2.elasticsearch-client"
            - name: cluster.initial_master_nodes
              value: "elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2"
            - name: ES_JAVA_OPTS
              value: -Xms512m -Xmx512m
            - name: node.master
              value: "false"
            - name: node.ingest
              value: "true"
            - name: node.data
              value: "false"
            - name: cluster.name
              value: "elasticsearch"
            - name: node.name
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: xpack.security.enabled
              value: "true"
            - name: xpack.security.transport.ssl.enabled
              value: "true"
            - name: xpack.monitoring.collection.enabled
              value: "true"
            - name: xpack.security.transport.ssl.verification_mode
              value: "certificate"
            - name: xpack.security.transport.ssl.keystore.path
              value: "/usr/share/elasticsearch/config/elastic-certificates.p12"
            - name: xpack.security.transport.ssl.truststore.path
              value: "/usr/share/elasticsearch/config/elastic-certificates.p12"

          volumeMounts:
           - mountPath: /usr/share/elasticsearch/data
             name: pv-storage-elastic-client
           - name: elastic-certificates
             readOnly: true
             mountPath: "/usr/share/elasticsearch/config/elastic-certificates.p12"
             subPath: elastic-certificates.p12
           - mountPath: /etc/localtime
             name: localtime

          securityContext:
            privileged: true

      volumes:
      - name: elastic-certificates
        secret:
          secretName: elastic-certificates
      - hostPath:
          path: /etc/localtime
        name: localtime

  volumeClaimTemplates:
  - metadata:
      name: pv-storage-elastic-client
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-storage"
      resources:
        requests:
          storage: 1Gi

4、创建ES集群的Service：

[root@master1 elk]# cat 04-es-service.yaml 
apiVersion: v1
kind: Service
metadata:
  namespace: ns-elk
  name: elasticsearch-master
  labels:
    app: elasticsearch
    role: master
spec:
  selector:
    app: elasticsearch
    role: master
  type: NodePort
  ports:
  - port: 9200
    nodePort: 30001
    targetPort: 9200
---
apiVersion: v1
kind: Service
metadata:
  namespace: ns-elk
  name: elasticsearch-data
  labels:
    app: elasticsearch
    role: data
spec:
  selector:
    app: elasticsearch
    role: data
  type: NodePort
  ports:
  - port: 9200
    nodePort: 30002
    targetPort: 9200
---
apiVersion: v1
kind: Service
metadata:
  namespace: ns-elk
  name: elasticsearch-client
  labels:
    app: elasticsearch
    role: client
spec:
  selector:
    app: elasticsearch
    role: client
  type: NodePort
  ports:
  - port: 9200
    nodePort: 30003
    targetPort: 9200

5、设置ES集群密码

[root@master1 elk]# kubectl -n ns-elk exec -it $(kubectl -n ns-elk get pods | grep elasticsearch-master | sed -n 1p | awk '{print $1}') -- bin/elasticsearch-setup-passwords auto -b
Changed password for user apm_system
PASSWORD apm_system = mdxoGjVA76RDvX5d5DCC

Changed password for user kibana_system
PASSWORD kibana_system = VQKC8RLuziSVagRNv7cg

Changed password for user kibana
PASSWORD kibana = VQKC8RLuziSVagRNv7cg

Changed password for user logstash_system
PASSWORD logstash_system = EH1xXzbkD5g3ZhyqZUjP

Changed password for user beats_system
PASSWORD beats_system = UJzPnlmujU2eas0nJeC3

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = dvCwt1HNHfyCiVjtLr7s

Changed password for user elastic
PASSWORD elastic = lGTiRY1ZcChmlNpr5AFX

牢记上面的密码！！！

四、部署Kibana

// 使用上面生成的用户 elastic 的密码 03sWFWzGOjNOCioqcbV3（登录 kibana 要用这个用户和密码）

[root@master1 elk]# kubectl -n ns-elk create secret generic elasticsearch-password --from-literal password=03sWFWzGOjNOCioqcbV3

部署kibana

[root@master1 elk]# cat 05-kibana.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: ns-elk
  name: kibana-config
  labels:
    app: kibana
data:
  kibana.yml: |-
    server.host: 0.0.0.0
    elasticsearch:
      hosts: ${ELASTICSEARCH_HOSTS}
      username: ${ELASTICSEARCH_USER}
      password: ${ELASTICSEARCH_PASSWORD}
---
kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    app: kibana
  name: kibana
  namespace: ns-elk
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      nodeSelector:
        node: node2
      containers:
        - name: kibana
          image: kibana:7.16.2
          ports:
            - containerPort: 5601
              protocol: TCP
          env:
            - name: SERVER_PUBLICBASEURL
              value: "http://0.0.0.0:5601"
            - name: I18N.LOCALE
              value: zh-CN
            - name: ELASTICSEARCH_HOSTS
              value: "http://elasticsearch-client:9200"
            - name: ELASTICSEARCH_USER
              value: "elastic"
            - name: ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: elasticsearch-password
                  key: password
            - name: xpack.encryptedSavedObjects.encryptionKey
              value: "min-32-byte-long-strong-encryption-key"

          volumeMounts:
          - name: kibana-config
            mountPath: /usr/share/kibana/config/kibana.yml
            readOnly: true
            subPath: kibana.yml
          - mountPath: /etc/localtime
            name: localtime
      volumes:
      - name: kibana-config
        configMap:
          name: kibana-config
      - hostPath:
          path: /etc/localtime
        name: localtime
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: kibana
  name: kibana-service
  namespace: ns-elk
spec:
  ports:
  - port: 5601
    targetPort: 5601
    nodePort: 30004
  type: NodePort
  selector:
    app: kibana

五、部署zookeeper/kafka集群

1、部署zookeeper

（1）创建zookeeper的PV

[root@master1 elk]# cat 00-pv-zk.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-zk-0
  namespace: ns-elk
  labels:
    name: local-storage-zk-0
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /elk/zookeeper
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master1
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-zk-1
  namespace: ns-elk
  labels:
    name: local-storage-zk-1
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /elk/zookeeper
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master2
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-zk-2
  namespace: ns-elk
  labels:
    name: local-storage-zk-2
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /elk/zookeeper
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master3

（2）部署zookeeper

[root@master1 elk]# cat 06-zk.yaml 
apiVersion: v1
kind: Service
metadata:
  name: zookeeper
  namespace: ns-elk
  labels:
    app: zookeeper
spec:
  type: NodePort
  ports:
  - port: 2181
    nodePort: 30005
    targetPort: 2181
  selector:
    app: zookeeper
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: zookeeper-pdb
  namespace: ns-elk
spec:
  selector:
    matchLabels:
      app: zookeeper
  minAvailable: 2
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zookeeper
  namespace: ns-elk
spec:
  selector:
    matchLabels:
      app: zookeeper
  serviceName: zookeeper
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
      - name: kubernetes-zookeeper
        imagePullPolicy: IfNotPresent
        image: "mirrorgooglecontainers/kubernetes-zookeeper:1.0-3.4.10"
        ports:
        - containerPort: 2181
          name: client
        - containerPort: 2888
          name: server
        - containerPort: 3888
          name: leader-election
        command:
        - sh
        - -c
        - "start-zookeeper \
          --servers=3 \
          --data_dir=/var/lib/zookeeper/data \
          --data_log_dir=/var/lib/zookeeper/data/log \
          --conf_dir=/opt/zookeeper/conf \
          --client_port=2181 \
          --election_port=3888 \
          --server_port=2888 \
          --tick_time=2000 \
          --init_limit=10 \
          --sync_limit=5 \
          --heap=512M \
          --max_client_cnxns=60 \
          --snap_retain_count=3 \
          --purge_interval=12 \
          --max_session_timeout=40000 \
          --min_session_timeout=4000 \
          --log_level=INFO"
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - "zookeeper-ready 2181"
          initialDelaySeconds: 10
          timeoutSeconds: 5
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - "zookeeper-ready 2181"
          initialDelaySeconds: 10
          timeoutSeconds: 5
        volumeMounts:
        - name: zookeeper
          mountPath: /var/lib/zookeeper
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
  volumeClaimTemplates:
  - metadata:
      name: zookeeper
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-storage"
      resources:
        requests:
          storage: 1Gi

2、部署kafka

（1）创建kafka的PV

[root@master1 elk]# cat 00-pv-kafka.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-kafka-0
  namespace: ns-elk
  labels:
    name: local-storage-kafka-0
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /elk/kafka
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master1
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-kafka-1
  namespace: ns-elk
  labels:
    name: local-storage-kafka-1
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /elk/kafka
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master2
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-kafka-2
  namespace: ns-elk
  labels:
    name: local-storage-kafka-2
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /elk/kafka
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master3

（2）部署kafka

[root@master1 elk]# cat 07-kafka.yaml 
---
apiVersion: v1
kind: Service
metadata:
  name: kafka
  namespace: ns-elk
  labels:
    app: kafka
spec:
  type: NodePort
  ports:
  - port: 9092
    nodePort: 30006
    targetPort: 9092
  selector:
    app: kafka
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: kafka-pdb
  namespace: ns-elk
spec:
  selector:
    matchLabels:
      app: kafka
  maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: ns-elk
spec:
  selector:
     matchLabels:
        app: kafka
  serviceName: kafka
  replicas: 3
  template:
    metadata:
      labels:
        app: kafka
    spec:
      terminationGracePeriodSeconds: 300
      containers:
      - name: k8s-kafka
        imagePullPolicy: IfNotPresent
        image: fastop/kafka:2.2.0
        resources:
          requests:
            memory: "600Mi"
            cpu: 500m
        ports:
        - containerPort: 9092
          name: server
        command:
        - sh
        - -c
        - "exec kafka-server-start.sh /opt/kafka/config/server.properties --override broker.id=${HOSTNAME##*-} \
          --override listeners=PLAINTEXT://:9092 \
          --override zookeeper.connect=zookeeper.ns-elk.svc.cluster.local:2181 \
          --override log.dir=/var/lib/kafka \
          --override auto.create.topics.enable=true \
          --override auto.leader.rebalance.enable=true \
          --override background.threads=10 \
          --override compression.type=producer \
          --override delete.topic.enable=false \
          --override leader.imbalance.check.interval.seconds=300 \
          --override leader.imbalance.per.broker.percentage=10 \
          --override log.flush.interval.messages=9223372036854775807 \
          --override log.flush.offset.checkpoint.interval.ms=60000 \
          --override log.flush.scheduler.interval.ms=9223372036854775807 \
          --override log.retention.bytes=-1 \
          --override log.retention.hours=168 \
          --override log.roll.hours=168 \
          --override log.roll.jitter.hours=0 \
          --override log.segment.bytes=1073741824 \
          --override log.segment.delete.delay.ms=60000 \
          --override message.max.bytes=1000012 \
          --override min.insync.replicas=1 \
          --override num.io.threads=8 \
          --override num.network.threads=3 \
          --override num.recovery.threads.per.data.dir=1 \
          --override num.replica.fetchers=1 \
          --override offset.metadata.max.bytes=4096 \
          --override offsets.commit.required.acks=-1 \
          --override offsets.commit.timeout.ms=5000 \
          --override offsets.load.buffer.size=5242880 \
          --override offsets.retention.check.interval.ms=600000 \
          --override offsets.retention.minutes=1440 \
          --override offsets.topic.compression.codec=0 \
          --override offsets.topic.num.partitions=50 \
          --override offsets.topic.replication.factor=3 \
          --override offsets.topic.segment.bytes=104857600 \
          --override queued.max.requests=500 \
          --override quota.consumer.default=9223372036854775807 \
          --override quota.producer.default=9223372036854775807 \
          --override replica.fetch.min.bytes=1 \
          --override replica.fetch.wait.max.ms=500 \
          --override replica.high.watermark.checkpoint.interval.ms=5000 \
          --override replica.lag.time.max.ms=10000 \
          --override replica.socket.receive.buffer.bytes=65536 \
          --override replica.socket.timeout.ms=30000 \
          --override request.timeout.ms=30000 \
          --override socket.receive.buffer.bytes=102400 \
          --override socket.request.max.bytes=104857600 \
          --override socket.send.buffer.bytes=102400 \
          --override unclean.leader.election.enable=true \
          --override zookeeper.session.timeout.ms=6000 \
          --override zookeeper.set.acl=false \
          --override broker.id.generation.enable=true \
          --override connections.max.idle.ms=600000 \
          --override controlled.shutdown.enable=true \
          --override controlled.shutdown.max.retries=3 \
          --override controlled.shutdown.retry.backoff.ms=5000 \
          --override controller.socket.timeout.ms=30000 \
          --override default.replication.factor=1 \
          --override fetch.purgatory.purge.interval.requests=1000 \
          --override group.max.session.timeout.ms=300000 \
          --override group.min.session.timeout.ms=6000 \
          --override inter.broker.protocol.version=2.2.0 \
          --override log.cleaner.backoff.ms=15000 \
          --override log.cleaner.dedupe.buffer.size=134217728 \
          --override log.cleaner.delete.retention.ms=86400000 \
          --override log.cleaner.enable=true \
          --override log.cleaner.io.buffer.load.factor=0.9 \
          --override log.cleaner.io.buffer.size=524288 \
          --override log.cleaner.io.max.bytes.per.second=1.7976931348623157E308 \
          --override log.cleaner.min.cleanable.ratio=0.5 \
          --override log.cleaner.min.compaction.lag.ms=0 \
          --override log.cleaner.threads=1 \
          --override log.cleanup.policy=delete \
          --override log.index.interval.bytes=4096 \
          --override log.index.size.max.bytes=10485760 \
          --override log.message.timestamp.difference.max.ms=9223372036854775807 \
          --override log.message.timestamp.type=CreateTime \
          --override log.preallocate=false \
          --override log.retention.check.interval.ms=300000 \
          --override max.connections.per.ip=2147483647 \
          --override num.partitions=4 \
          --override producer.purgatory.purge.interval.requests=1000 \
          --override replica.fetch.backoff.ms=1000 \
          --override replica.fetch.max.bytes=1048576 \
          --override replica.fetch.response.max.bytes=10485760 \
          --override reserved.broker.max.id=1000 "
        env:
        - name: KAFKA_HEAP_OPTS
          value : "-Xmx512M -Xms512M"
        - name: KAFKA_OPTS
          value: "-Dlogging.level=INFO"
        volumeMounts:
        - name: kafka
          mountPath: /var/lib/kafka
        readinessProbe:
          tcpSocket:
            port: 9092
          timeoutSeconds: 1
          initialDelaySeconds: 5
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
  volumeClaimTemplates:
  - metadata:
      name: kafka
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-storage"
      resources:
        requests:
          storage: 1Gi

3、测试kafka

（1）通过zookeeper查看broker

[root@master1 elk]# kubectl exec -it zookeeper-1 -n ns-elk bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
zookeeper@zookeeper-1:/$  zkCli.sh 
Connecting to localhost:2181

[zk: localhost:2181(CONNECTED) 0] get /brokers/ids/0
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://kafka-0.kafka.ns-elk.svc.cluster.local:9093"],"jmx_port":-1,"host":"kafka-0.kafka.ns-elk.svc.cluster.local","timestamp":"1641887271398","port":9093,"version":4}
cZxid = 0x200000024
ctime = Tue Jan 11 07:47:51 UTC 2022
mZxid = 0x200000024
mtime = Tue Jan 11 07:47:51 UTC 2022
pZxid = 0x200000024
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x27e480b276e0001
dataLength = 246
numChildren = 0
[zk: localhost:2181(CONNECTED) 1] get /brokers/ids/1
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://kafka-1.kafka.ns-elk.svc.cluster.local:9093"],"jmx_port":-1,"host":"kafka-1.kafka.ns-elk.svc.cluster.local","timestamp":"1641887242316","port":9093,"version":4}
cZxid = 0x20000001e
ctime = Tue Jan 11 07:47:22 UTC 2022
mZxid = 0x20000001e
mtime = Tue Jan 11 07:47:22 UTC 2022
pZxid = 0x20000001e
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x27e480b276e0000
dataLength = 246
numChildren = 0
[zk: localhost:2181(CONNECTED) 2] get /brokers/ids/2
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://kafka-2.kafka.ns-elk.svc.cluster.local:9093"],"jmx_port":-1,"host":"kafka-2.kafka.ns-elk.svc.cluster.local","timestamp":"1641888604437","port":9093,"version":4}
cZxid = 0x20000002d
ctime = Tue Jan 11 08:10:04 UTC 2022
mZxid = 0x20000002d
mtime = Tue Jan 11 08:10:04 UTC 2022
pZxid = 0x20000002d
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x27e480b276e0002
dataLength = 246
numChildren = 0

（2）kafka生产消费测试

创建topic

[root@master1 elk]# kubectl exec -it kafka-0 -n ns-elk sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
$ pwd
/
$ cd /opt/kafka/bin
$ ./kafka-topics.sh --create --topic test --zookeeper zookeeper.ns-elk.svc.cluster.local:2181 --partitions 3 --replication-factor 3    
Created topic "test".
$ ./kafka-topics.sh --list --zookeeper zookeeper.ns-elk.svc.cluster.local:2181
test

生产消息：

$ ./kafka-console-producer.sh --topic test --broker-list kafka-0.kafka.ns-elk.svc.cluster.local:9093   
111

另起一个终端消费消息：

$ ./kafka-console-consumer.sh --topic test --zookeeper zookeeper.ns-elk.svc.cluster.local:2181 --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
111

消费是正常的！

六、部署filebeat

我的filebeat使用DemoSet，并且是自动发现日志模式：

[root@master1 elk]# cat 08-filebeat.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: ns-elk
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      paths:
        - '/var/lib/docker/containers/*/*.log'
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/lib/docker/containers/"
    processors:
      - add_cloud_metadata:
      - add_host_metadata:
    output:
      kafka:
        enabled: true
        hosts: ["kafka-0.kafka.ns-elk.svc.cluster.local:9092","kafka-1.kafka.ns-elk.svc.cluster.local:9092","kafka-2.kafka.ns-elk.svc.cluster.local:9092"]
        topic: "filebeat"
        max_message_bytes: 5242880
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: ns-elk
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.16.2
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: ns-elk
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: filebeat
  namespace: ns-elk
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: ns-elk
roleRef:
  kind: Role
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: filebeat-kubeadm-config
  namespace: ns-elk
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: ns-elk
roleRef:
  kind: Role
  name: filebeat-kubeadm-config
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  - nodes
  verbs:
  - get
  - watch
  - list
- apiGroups: ["apps"]
  resources:
    - replicasets
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: filebeat
  namespace: ns-elk
  labels:
    k8s-app: filebeat
rules:
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs: ["get", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: filebeat-kubeadm-config
  namespace: ns-elk
  labels:
    k8s-app: filebeat
rules:
  - apiGroups: [""]
    resources:
      - configmaps
    resourceNames:
      - kubeadm-config
    verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: ns-elk
  labels:
    k8s-app: filebeat

七、部署Logstash

[root@master1 elk]# cat 09-logstash.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-configmap
  namespace: ns-elk
data:
  logstash.yml: |
    http.host: "0.0.0.0"
    path.config: /usr/share/logstash/pipeline
  logstash.conf: |
    input {
      kafka {
        bootstrap_servers => "kafka-0.kafka.ns-elk.svc.cluster.local:9092,kafka-1.kafka.ns-elk.svc.cluster.local:9092,kafka-2.kafka.ns-elk.svc.cluster.local:9092"
        topics => ["filebeat"]
        codec => "json"
      }
    }
    filter {
      date {
        match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
      }
    }
      output {
        elasticsearch {
          hosts => ["elasticsearch-client:9200"]
          user => "elastic"
          password => "lGTiRY1ZcChmlNpr5AFX"
          index => "kubernetes-%{+YYYY.MM.dd}"
      }
    }

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash-deployment
  namespace: ns-elk
spec:
  selector:
    matchLabels:
      app: logstash
  replicas: 1
  template:
    metadata:
      labels:
        app: logstash
    spec:
      nodeSelector:
        node: node2
      containers:
      - name: logstash
        image: docker.elastic.co/logstash/logstash:7.16.2
        ports:
        - containerPort: 5044
        volumeMounts:
          - name: config-volume
            mountPath: /usr/share/logstash/config
          - name: logstash-pipeline-volume
            mountPath: /usr/share/logstash/pipeline
          - mountPath: /etc/localtime
            name: localtime
      volumes:
      - name: config-volume
        configMap:
          name: logstash-configmap
          items:
            - key: logstash.yml
              path: logstash.yml
      - name: logstash-pipeline-volume
        configMap:
          name: logstash-configmap
          items:
            - key: logstash.conf
              path: logstash.conf
      - hostPath:
          path: /etc/localtime
        name: localtime
---
kind: Service
apiVersion: v1
metadata:
  name: logstash-service
  namespace: ns-elk
spec:
  selector:
    app: logstash
  type: NodePort
  ports:
  - protocol: TCP
    port: 5044
    targetPort: 5044
    nodePort: 30007

以上创建完成后，可以使用kubectl logs -f po-name -n ns-elk来查看应用的日志，检查应用是否正常运行。
下面是创建完成的资源清单：

[root@master1 elk]# kubectl get all -n ns-elk
NAME                                       READY   STATUS    RESTARTS   AGE
pod/elasticsearch-client-0                 1/1     Running   1          34h
pod/elasticsearch-client-1                 1/1     Running   1          34h
pod/elasticsearch-client-2                 1/1     Running   1          27h
pod/elasticsearch-data-0                   1/1     Running   1          34h
pod/elasticsearch-data-1                   1/1     Running   1          34h
pod/elasticsearch-data-2                   1/1     Running   1          34h
pod/elasticsearch-data-3                   1/1     Running   1          34h
pod/elasticsearch-data-4                   1/1     Running   1          27h
pod/elasticsearch-master-0                 1/1     Running   1          34h
pod/elasticsearch-master-1                 1/1     Running   1          34h
pod/elasticsearch-master-2                 1/1     Running   1          27h
pod/filebeat-bx9cd                         1/1     Running   0          14m
pod/filebeat-d8gt9                         1/1     Running   0          14m
pod/filebeat-gd5vk                         1/1     Running   0          14m
pod/filebeat-klkhz                         1/1     Running   0          14m
pod/filebeat-tgb4r                         1/1     Running   0          14m
pod/kafka-0                                1/1     Running   3          14m
pod/kafka-1                                1/1     Running   0          13m
pod/kafka-2                                1/1     Running   0          12m
pod/kibana-67887bfc64-w4gcf                1/1     Running   1          33h
pod/logstash-deployment-797f4dfc55-jlh2f   1/1     Running   0          6m31s
pod/zookeeper-0                            1/1     Running   0          14m
pod/zookeeper-1                            1/1     Running   0          14m
pod/zookeeper-2                            1/1     Running   0          14m

NAME                           TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/elasticsearch-client   NodePort   10.99.153.13     <none>        9200:30003/TCP   45h
service/elasticsearch-data     NodePort   10.109.197.124   <none>        9200:30002/TCP   45h
service/elasticsearch-master   NodePort   10.99.142.132    <none>        9200:30001/TCP   45h
service/kafka                  NodePort   10.102.5.188     <none>        9092:30006/TCP   14m
service/kibana-service         NodePort   10.107.128.193   <none>        5601:30004/TCP   33h
service/logstash-service       NodePort   10.104.7.238     <none>        5044:30007/TCP   6m24s
service/zookeeper              NodePort   10.105.24.169    <none>        2181:30005/TCP   14m

NAME                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/filebeat   5         5         5       5            5           <none>          14m

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kibana                1/1     1            1           33h
deployment.apps/logstash-deployment   1/1     1            1           6m31s

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/kibana-67887bfc64                1         1         1       33h
replicaset.apps/logstash-deployment-797f4dfc55   1         1         1       6m31s

NAME                                    READY   AGE
statefulset.apps/elasticsearch-client   3/3     34h
statefulset.apps/elasticsearch-data     5/5     44h
statefulset.apps/elasticsearch-master   3/3     44h
statefulset.apps/kafka                  3/3     14m
statefulset.apps/zookeeper              3/3     15m

可以看到，所有的资源都是正常运行的。这时候我们打开kibana页面，检查索引是否创建：
在这里插入图片描述
创建索引模式，看一下数据：

日志数据也有了，整个集群基本搭建完成！

Dustin.Hoffman

关注

5
点赞
踩
26

收藏

觉得还不错? 一键收藏
打赏
10
评论
kubernetes上部署ELK集群

一、系统架构IP角色主机名192.168.1.3mastermaster1192.168.1.4mastermaster2192.168.1.5mastermaster3192.168.1.6nodenode1192.168.1.7nodenode2二、创建StorageClass和Namespace1、创建StorageClass[root@master1 ~]# mkdir -p /elk/{master,data,clien
复制链接

扫一扫