k8s部署Elasticsearch集群+Kibana方案--开启X-Pack 安全认证

前言

本文中使用StatefulSet 方式部署 Elasticsearch 集群，并且开启X-Pack 安全认证，存储使用的是NFS，属于一个初学者自己探索的方案，如果有比较好的方案，还请不吝评论赐教。
版本说明：

• Kubernetes v1.25.6 – v1.26.4
• Elasticsearch, Kibana 7.13.3
• NFS Subdir External Provisioner

前置环境
需要安装好Kubernetes 集群，并且配置了存储类(StorageClass)。

一、部署Elasticsearch集群

1. 创建配置文件

elastic-worker-ns.yaml

apiVersion: v1           # 创建命名空间
kind: Namespace
metadata:
  labels:
    app: es7-cluster
    kubernetes.io/name: "Elasticsearch"
  name: elastic-worker

es7-cluster.yaml

---
apiVersion: v1             # 创建service 文件用于内部通讯
kind: Service
metadata:
  name: es7-headless
  namespace: elastic-worker
  labels:
    app: es7-cluster
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "Elasticsearch"
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  ports:
  - name: rest-api
    port: 9200
    targetPort: 9200
  - name: inter-node
    port: 9300
    targetPort: 9300
  selector:
    app: es7-cluster
---
apiVersion: v1             
kind: ServiceAccount
metadata:
  name: es7-cluster
  namespace: elastic-worker
  labels:
    app: es7-cluster
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: es7-cluster
  namespace: elastic-worker
  labels:
    app: es7-cluster
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - "services"
  - "namespaces"
  - "endpoints"
  verbs:
  - "get"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: es7-cluster
  namespace: elastic-worker
  labels:
    app: es7-cluster
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
  name: es7-cluster
  namespace: kube-system
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: es7-cluster
  apiGroup: ""
---
apiVersion: apps/v1             # 创建有状态的服务
kind: StatefulSet
metadata:
  name: es7-cluster
  namespace: elastic-worker
  labels:
    app: es7-cluster
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    srv: srv-elasticsearch
spec:
  serviceName: es7-headless
  replicas: 3
  selector:
    matchLabels:
      app: es7-cluster
      kubernetes.io/cluster-service: "true"
  template:
    metadata:
      labels:
        app: es7-cluster
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccountName: es7-cluster
      containers:         # 主容器
      - name: elasticsearch
        image: registry.cn-hangzhou.aliyuncs.com/greatmap/elasticsearch:7.13.3-p12    # 自定义镜像，下边会详细说明
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9200
          name: rest-api
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        - name: localtime
          readOnly: true
          mountPath: /etc/localtime
        env:
        - name: cluster.name
          value: "es7-cluster"
        - name: node.name
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: discovery.zen.minimum_master_nodes
          value: "2"
        - name: discovery.seed_hosts
          value: "es7-headless"
        - name: cluster.initial_master_nodes
          value: "es7-cluster-0,es7-cluster-1,es7-cluster-2"
        - name: ES_JAVA_OPTS
          value: "-Xms1g -Xmx1g"
        - name: xpack.security.enabled    # X-Pack 安全认证
          value: "true"
        - name: xpack.security.transport.ssl.enabled
          value: "true"
        - name: xpack.security.transport.ssl.verification_mode  # 证书校验类型
          value: "certificate"
        - name: xpack.security.transport.ssl.keystore.path    # 证书路径
          value: "elastic-certificates.p12"
        - name: xpack.security.transport.ssl.truststore.path
          value: "elastic-certificates.p12"
        - name: xpack.monitoring.ui.container.elasticsearch.enabled   # 生成并提供与容器相关的监控数据，待验证
          value: "true"
        #- name: reindex.remote.whitelist        # 设置同步白名单，可以用来数据迁移
        #  value: "192.168.10.13:9200"
      initContainers:        # 初始化容器
      - name: fix-permissions
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        - name: localtime
          readOnly: true
          mountPath: /etc/localtime
      - name: increase-vm-max-map
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["sh", "-c", "ulimit -n 65536"]
      volumes:
      - name: localtime
        hostPath:
          path: /etc/localtime
          type: ''
  volumeClaimTemplates:   
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteMany" ]
      # 使用的存储类名称，需要配置一个有效的存储类
      storageClassName: "managed-nfs-storage"
      resources:
        requests:
          storage: 2Gi

2. 部署Elasticsearch集群

kubectl create -f es7-cluster.yaml

3. 相关说明

• 单节点模式只开启xpack.security.enabled 既可以，但是集群模式通信就需要用到证书；
• 只开启xpack.security.enabled 在重新apply 时会失败，提示xpack.security.transport.ssl.enabled 为 false 请设置为 true；
• 再开启xpack.security.transport.ssl.enabled 需要生成且配置证书路径，但是查找相关资料都是启动后生成证书，再启动容器；在k8s 中 pod 会无限重启，没办法生成证书；
• 顾此自定义镜像将证书直接打到镜像里，使用Dockerfile进行构建
• 证书校验类型：full、certificate、none；默认值为“ full ”；

如何生成证书：

# 使用docker 运行elasticsearch容器
docker run -it --rm elasticsearch:7.13.3  bash
# 生成证书
./bin/elasticsearch-certutil cert -out config/elastic-certificates.p12 -pass ""

将证书保存到本地，再通过Dockerfile重新封装镜像

FROM elasticsearch:7.13.3
LABEL maintainer="YZEQIANG <yinzeqiang66@126.com>"
COPY elastic-certificates.p12 /usr/share/elasticsearch/config/
RUN  chown 1000:0 /usr/share/elasticsearch/config/elastic-certificates.p12
EXPOSE 9200 9300
CMD ["eswrapper"]

阿里云仓库：(支持x86_64、arm64)

docker pull registry.cn-hangzhou.aliyuncs.com/greatmap/elasticsearch:7.13.3-p12

4. 设置密码

进入到任意节点

kubectl exec -it -n elastic-worker pods/es7-cluster-1  -- bash

./bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y

# interactive 手动设置密码（Stack123），如果是auto，自动随机生成密码
Enter password for [elastic]: 
Reenter password for [elastic]: 
Enter password for [apm_system]: 
Reenter password for [apm_system]: 
Enter password for [kibana_system]: 
Reenter password for [kibana_system]: 
Enter password for [logstash_system]: 
Reenter password for [logstash_system]: 
Enter password for [beats_system]: 
Reenter password for [beats_system]: 
Enter password for [remote_monitoring_user]: 
Reenter password for [remote_monitoring_user]: 

5. 将es集群对外暴露

es7-external.yaml

apiVersion: v1             # 创建service 文件用于内部通讯
kind: Service
metadata:
  name: es7-external
  namespace: elastic-worker
  labels:
    app: es7-cluster
spec:
  type: NodePort
  ports:
  - name: rest-api
    port: 9200
    protocol: TCP
    targetPort: 9200
    nodePort: 30920
  selector:
    app: es7-cluster

6. 外部连接验证

使用的是edge扩展插件
https://microsoftedge.microsoft.com/addons/detail/elasticvue/geifniocjfnfilcbeloeidajlfmhdlgo?hl=zh-CN

二、部署Kibana

1. 创建配置文件

kibana.yaml

apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: elastic-worker
  labels:
    k8s-app: kibana
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "Kibana"
    srv: srv-kibana
spec:
  type: NodePort
  ports:
  - port: 5601
    nodePort: 30000
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: elastic-worker
  labels:
    k8s-app: kibana
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    srv: srv-kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana
  template:
    metadata:
      labels:
        k8s-app: kibana
    spec:
      containers:
      - name: kibana
        image: kibana:7.13.3
        volumeMounts:
        - name: kibana-config
          mountPath: /usr/share/kibana/config/kibana.yml
          readOnly: true
          subPath: kibana.yml
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP
      volumes:
      - name: kibana-config
        configMap:
          name: kibana
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana
  namespace: elastic-worker
  labels:
    k8s-app: kibana
data:
  kibana.yml: |-
  
    server.name: kibana
    server.host: "0.0.0.0"
    elasticsearch.hosts: [ "http://es7-headless:9200" ]
    elasticsearch.username: "elastic"
    elasticsearch.password: "Stack123"
    monitoring.ui.container.elasticsearch.enabled: true
    i18n.locale: "zh-CN"
    kibana.index: ".kibana"

2. 部署Kibana

kubectl apply -f kibana.yaml

3. 访问验证

http://ip:port

参考文档：

• ECK官方文档
• Kibana 将语言设置为中文
• ElasticSearch7.X启动X-Pack
• k8s 部署ELK并设置用户名密码
• Kubernetes 之 StorageClass 资源(NFS)
• k8s 部署生产级 elasticsearch+kibana 步骤
• k8s部署 elfk 7.x + X-Pack，使用ConfigMap方式