ELK学习笔记（一）——使用K8S部署ElasticSearch8.15.0集群

一、下载镜像

#1、下载官方镜像
docker pull elasticsearch:8.15.0
#2、打新tag
docker tag elasticsearch:8.15.0 192.168.9.41:8088/new-erp-common/elasticsearch:8.15.0
#3、推送到私有仓库harbor
docker push 192.168.9.41:8088/new-erp-common/elasticsearch:8.15.0

二、优化宿主机配置

部署elasticsearch集群需要先优化宿主机(所有k8s节点都要优化,不优化会部署失败)

vim /etc/sysctl.conf 

vm.max_map_count=262144 # (用于设置 Linux 系统内核中允许用户态程序的最大内存区域数量。通过设置该参数，可以控制系统中允许映射的内存区域的最大数量，这对于一些需要大量内存映射的应用程序是很有用)

sysctl -p #生效配置

三、创建工作目录

下面是搭建ES的目录一览图

四、准备yaml配置文件

4.1准备ConfigMap配置

创建ConfigMap配置，里面主要配置了elasticsearch.yml需要的配置

$ cat config-map-es.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-map-es
  namespace: renpho-erp-common
data:
  # 下面2行不确定可不可以删除，但因为在elasticsearch.yml中配置了，所以感觉这2行可以删了
  network.host: "0.0.0.0"
  cluster.name: "es-cluster"
  elasticsearch.yml: |
    #设置集群名称
    cluster.name: es-cluster
    #设置网络访问节点【其他节点修改项】
    network.host: "0.0.0.0"
    #设置网络访问端口
    http.port: 9200
    transport.port: 9300
    node.roles: [ingest,master,data]
    #节点发现
    discovery.seed_hosts: ["elasticsearch-0.elasticsearch.renpho-erp-common.svc.cluster.local","elasticsearch-1.elasticsearch.renpho-erp-common.svc.cluster.local","elasticsearch-2.elasticsearch.renpho-erp-common.svc.cluster.local"]
    #初始化集群
    cluster.initial_master_nodes: ["elasticsearch-0","elasticsearch-1","elasticsearch-2"]
    #启用安全
    xpack.security.enabled: true
    xpack.security.enrollment.enabled: true

    #客户端连接加密
    xpack.security.http.ssl:
      enabled: true
      keystore.path: /usr/share/elasticsearch/config/local-certs/http.p12
      truststore.path: /usr/share/elasticsearch/config/local-certs/http.p12
    #集群内节点连接加密
    xpack.security.transport.ssl:
      enabled: true
      verification_mode: certificate
      keystore.path: /usr/share/elasticsearch/config/local-certs/elastic-certificates.p12
      truststore.path: /usr/share/elasticsearch/config/local-certs/elastic-certificates.p12

    #必须set为true，否则kibana报错
    search.allow_expensive_queries: true
    #禁用geoip下载
    ingest.geoip.downloader.enabled: false

4.2准备Service及StatefulSet文件

在 Kubernetes 上搭建 Elasticsearch 集群时，通常会创建两个不同类型的 Service：一个无头服务（Headless Service）和一个普通的有头服务（ClusterIP Service）。这两种服务各有其特定的用途和作用，以下是它们的具体用途和原因：

a. 无头服务（Headless Service）

• 作用： 无头服务是通过设置 clusterIP: None 创建的，这意味着它不会分配一个单一的集群IP地址。这种服务不会进行负载均衡，而是直接暴露其背后所有的 Pod。
• 用途： 在 Elasticsearch 集群中，无头服务通常用于节点发现和集群的状态维护。通过无头服务，每个 Elasticsearch 节点可以获得集群中其他节点的 IP 地址，从而进行节点间的通信和发现。
• DNS 记录： 无头服务会为每个 Pod 创建一个独立的 DNS A 记录，这样，Elasticsearch 可以通过这些记录直接访问到其他节点。例如，如果有一个无头服务 elasticsearch-headless，且有三个 Pod（elasticsearch-0, elasticsearch-1, elasticsearch-2），这些 Pod 可以通过 DNS 名称 elasticsearch-0.elasticsearch-headless.namespace.svc.cluster.local 访问彼此。

b. 有头服务（ClusterIP Service）

• 作用： 有头服务（通常称为 ClusterIP 服务）会分配一个集群内部 IP 地址，并为该地址上的端口提供负载均衡。这意味着，任何请求发送到该服务的 IP 地址时，会被分配到后端的某一个 Pod。
• 用途： 在 Elasticsearch 集群中，有头服务通常用于外部客户端的连接和访问，例如 Kibana 或其他使用者查询 Elasticsearch 数据的应用程序。它可以提供一个单一的访问点，从而简化外部应用的连接配置。
• 负载均衡： 通过有头服务，Kubernetes 可以在多个 Elasticsearch 节点之间进行负载均衡，确保请求均匀分布，从而提高查询性能和服务的可用性。

c. 总结

• 无头服务：用于集群内的节点发现和通信，每个节点可以直接找到其他节点的 IP 地址，便于 Elasticsearch 集群中的主节点选举、数据复制和分片分配。
• 有头服务：用于提供一个稳定的、负载均衡的外部访问点，让外部应用或用户可以通过一个固定的服务 IP 地址来访问 Elasticsearch 集群，而无需关心背后 Pod 的具体 IP。

通过结合使用无头服务和有头服务，可以既保持集群内部节点间的灵活通信，又提供对外部客户端的统一访问接口，这是在 Kubernetes 上部署 Elasticsearch 集群的常见模式。

$ cat deploy-es2.yaml 
#无头服务
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: renpho-erp-common
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: db
    - port: 9300
      name: inter
---
#有头服务
apiVersion: v1
kind: Service
metadata:
  name: es-nodeport
  namespace: renpho-erp-common
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  type: NodePort
  ports:
    - port: 9200
      name: db
      nodePort: 30092
    - port: 9300
      name: inter
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: renpho-erp-common
  labels:
    app: elasticsearch
spec:
  podManagementPolicy: Parallel 
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: renpho.harbor.com/new-erp-common/elasticsearch:8.15.0
        imagePullPolicy: IfNotPresent
        securityContext: ##开启特权，因为要调整系统内核
          privileged: true
        resources:
          limits:
            cpu: 1
            memory: 2Gi
          requests:
            cpu: 0.5
            memory: 500Mi
       #command: ["/bin/sh","-c"]
       # args:
       # - |
       #   sleep 3600;
        env:
          - name: network.host
            valueFrom:
              configMapKeyRef:
                name: config-map-es
                key: network.host
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
        ports:
        - name: db
          containerPort: 9200
          protocol: TCP
        - name: inter
          containerPort: 9300
          protocol: TCP
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /usr/share/elasticsearch/data
          subPath: es-data
        - name: elasticsearch-data
          mountPath: /usr/share/elasticsearch/logs
          subPath: es-logs
        - name: elasticsearch-data
          mountPath: /usr/share/elasticsearch/.cache
          subPath: es-cache
        - name: elasticsearch-data
          mountPath: /usr/share/elasticsearch/plugins
          subPath: es-plugins
        - name: es-cert-file  #挂载存储目录
          mountPath: /usr/share/elasticsearch/config/local-certs
        - name: es-config  #挂载配置文件
          mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
          subPath: elasticsearch.yml
        - name: host-time  #挂载本地时区
          mountPath: /etc/localtime
          readOnly: true

      volumes:
      - name: es-config
        configMap:
          name: config-map-es
          defaultMode: 493 #文件权限为-rwxr-xr-x
      - name: es-cert-file
        secret:
          secretName: es-certificates
      - name: host-time
        hostPath: #挂载本地时区
          path: /etc/localtime
          type: ""
  volumeClaimTemplates:
  - metadata:
      name: elasticsearch-data
    spec:
      storageClassName: ssd-nfs-storage
      accessModes: [ "ReadWriteMany" ]
      resources:
        requests:
          storage: 50Gi

五、生成elastic集群所需的安全证书

为了方便生成证书，可以借助docker 运行 es 容器，但后进入容器内将证书生成好之后，再拷贝到宿主机备用。

##启动es容器
docker run -it -d --name es 192.168.9.41:8088/new-erp-common/elasticsearch:8.15.0
##进入容器生成证书
docker exec -it es bash

elasticsearch@62d07cf8df10:~$ pwd
/usr/share/elasticsearch

5.1生成CA证书

elasticsearch@62d07cf8df10:~$ ./bin/elasticsearch-certutil ca

这里可以选择添加证书密码，如果添加密码的话，后续使用CA证书去生成其他证书都需要先校验密码
默认会在当前目录下（/usr/share/elasticsearch）生成 elastic-stack-ca.p12 这个证书文件，在实际操作中根据自己的实际情况进行调整

5.2使用CA证书生成 transport证书

elasticsearch@62d07cf8df10:~$ ./bin/elasticsearch-certutil cert --ca /usr/share/elasticsearch/elastic-stack-ca.p12

最终会生成1个elastic-certificates.p12的证书文件

5.3使用CA证书生成http证书

elasticsearch@62d07cf8df10:~$ ./bin/elasticsearch-certutil http

## Elasticsearch HTTP Certificate Utility

The 'http' command guides you through the process of generating certificates
for use on the HTTP (Rest) interface for Elasticsearch.

This tool will ask you a number of questions in order to generate the right
set of files for your needs.

## Do you wish to generate a Certificate Signing Request (CSR)?

A CSR is used when you want your certificate to be created by an existing
Certificate Authority (CA) that you do not control (that is, you don't have
access to the keys for that CA). 

If you are in a corporate environment with a central security team, then you
may have an existing Corporate CA that can generate your certificate for you.
Infrastructure within your organisation may already be configured to trust this
CA, so it may be easier for clients to connect to Elasticsearch if you use a
CSR and send that request to the team that controls your CA.

If you choose not to generate a CSR, this tool will generate a new certificate
for you. That certificate will be signed by a CA under your control. This is a
quick and easy way to secure your cluster with TLS, but you will need to
configure all your clients to trust that custom CA.
#是否需要证书认证请求，选n
Generate a CSR? [y/N]n

## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?

If you have an existing CA certificate and key, then you can use that CA to
sign your new http certificate. This allows you to use the same CA across
multiple Elasticsearch clusters which can make it easier to configure clients,
and may be easier for you to manage.

If you do not have an existing CA, one will be generated for you.
#是否需要选择已存在得证书，选y
Use an existing CA? [y/N]y

## What is the path to your CA?

Please enter the full pathname to the Certificate Authority that you wish to
use for signing your new http certificate. This can be in PKCS#12 (.p12), JKS
(.jks) or PEM (.crt, .key, .pem) format.
#填入已存在ca证书路径
CA Path: /usr/share/elasticsearch/elastic-stack-ca.p12
Reading a PKCS12 keystore requires a password.
It is possible for the keystore's password to be blank,
in which case you can simply press <ENTER> at the prompt
#输入已存在证书密码，没有的话直接回车
Password for elastic-stack-ca.p12:

## How long should your certificates be valid?

Every certificate has an expiry date. When the expiry date is reached clients
will stop trusting your certificate and TLS connections will fail.

Best practice suggests that you should either:
(a) set this to a short duration (90 - 120 days) and have automatic processes
to generate a new certificate before the old one expires, or
(b) set it to a longer duration (3 - 5 years) and then perform a manual update
a few months before it expires.

You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D)
#证书有效时间
For how long should your certificate be valid? [5y] 7y

## Do you wish to generate one certificate per node?

If you have multiple nodes in your cluster, then you may choose to generate a
separate certificate for each of these nodes. Each certificate will have its
own private key, and will be issued for a specific hostname or IP address.

Alternatively, you may wish to generate a single certificate that is valid
across all the hostnames or addresses in your cluster.

If all of your nodes will be accessed through a single domain
(e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it
simpler to generate one certificate with a wildcard hostname (*.es.example.com)
and use that across all of your nodes.

However, if you do not have a common domain name, and you expect to add
additional nodes to your cluster in the future, then you should generate a
certificate per node so that you can more easily generate new certificates when
you provision new nodes.
#是否每个节点都需要生成，选n，所有节点共用一个
Generate a certificate per node? [y/N]n

## Which hostnames will be used to connect to your nodes?

These hostnames will be added as "DNS" names in the "Subject Alternative Name"
(SAN) field in your certificate.

You should list every hostname and variant that people will use to connect to
your cluster over http.
Do not list IP addresses here, you will be asked to enter them later.

If you wish to use a wildcard certificate (for example *.es.example.com) you
can enter that here.

Enter all the hostnames that you need, one per line.

When you are done, press <ENTER> once more to move on to the next step.
#输入集群所有节点主机名
#使用 Kubernetes 中 Pod 的 DNS 名称,可以避免 Pod IP 变化带来的问题。
#DNS 名称通常是 <pod-name>.<service-name>.<namespace>.svc.cluster.local 形式的
*.elasticsearch.renpho-erp-common.svc.cluster.local
elasticsearch.renpho-erp-common.svc.cluster.local
elasticsearch.renpho-erp-common
elasticsearch

You entered the following hostnames.

 - *.elasticsearch.renpho-erp-common.svc.cluster.local
 - elasticsearch.renpho-erp-common.svc.cluster.local
 - elasticsearch.renpho-erp-common
 - elasticsearch

#是否正确，选y
Is this correct [Y/n]y

#输入集群所有节点ip地址，由于上面使用的是DNS名称，所以这一步不用再输入固定IP地址，直接回车
## Which IP addresses will be used to connect to your nodes?

If your clients will ever connect to your nodes by numeric IP address, then you
can list these as valid IP "Subject Alternative Name" (SAN) fields in your
certificate.

If you do not have fixed IP addresses, or not wish to support direct IP access
to your cluster then you can just press <ENTER> to skip this step.

Enter all the IP addresses that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.


You did not enter any IP addresses.

#是否正确，选y
Is this correct [Y/n]y

## Other certificate options

The generated certificate will have the following additional configuration
values. These values have been selected based on a combination of the
information you have provided above and secure defaults. You should not need to
change these values unless you have specific requirements.

Key Name: elasticsearch-0.elasticsearch.renpho-erp-common.svc.cluster.local
Subject DN: CN=elasticsearch-0, DC=elasticsearch, DC=renpho-erp-common, DC=svc, DC=cluster, DC=local
Key Size: 2048
#是否修改证书配置，选n
Do you wish to change any of these options? [y/N]n

#输入密码,不想设置密码直接回车。建议为空，省点麻烦，这么多证书认证已经够够的了
## What password do you want for your private key(s)?

Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12".
This type of keystore is always password protected, but it is possible to use a
blank password.

If you wish to use a blank password, simply press <enter> at the prompt below.
Provide a password for the "http.p12" file:  [<ENTER> for none]

## Where should we save the generated files?

A number of files will be generated including your private key(s),
public certificate(s), and sample configuration options for Elastic Stack products.

These files will be included in a single zip archive.

What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip] 
#证书文件保存位置
Zip file written to /usr/share/elasticsearch/elasticsearch-ssl-http.zip


#解压缩刚生成得证书zip文件
elasticsearch@62d07cf8df10:~$ unzip elasticsearch-ssl-http.zip
Archive:  elasticsearch-ssl-http.zip
   creating: elasticsearch/
  inflating: elasticsearch/README.txt
  inflating: elasticsearch/http.p12
...

以上完成后将在/usr/share/elasticsearch下生成一个zip压缩文件。 解压文件，生成一个文件夹，里面包含两个文件夹:
elasticsearch文件夹包含http.p12及elasticsearch.yml的配置参考;
kibana文件夹包含elasticsearch-ca.pem及kibana.yml的配置参考;

5.4生成kibana使用的安全证书

为了方便起见，顺便将kibana使用的安全证书也一起生成。除了之前生成http证书时生成的 elasticsearch-ca.pem 之外还有3个文件

kibana.crt，kibana.key，kibana.csr
下面先生成 kibana.csr，kibana.key

elasticsearch@62d07cf8df10:~$ /usr/share/elasticsearch/bin/elasticsearch-certutil csr -name kibana -dns *.elasticsearch.renpho-erp-common.svc.cluster.local -dns elasticsearch.renpho-erp-common.svc.cluster.local -dns elasticsearch.renpho-erp-common -dns elasticsearch
#执行后默认会生成 csr-bundle.zip
#解压缩后得到kibana.csr ，kibana.key，用它2生成 kibana.crt
elasticsearch@62d07cf8df10:~$ unzip csr-bundle.zip 
Archive:  csr-bundle.zip
   creating: kibana/
  inflating: kibana/kibana.csr       
  inflating: kibana/kibana.key
# 生成crt文件
elasticsearch@62d07cf8df10:~$ cd kibana/
elasticsearch@62d07cf8df10:~$ openssl x509 -req -in kibana.csr -signkey kibana.key -out kibana.crt
Signature ok
subject=CN = kibana
Getting Private key
elasticsearch@62d07cf8df10:~$ ls -l
total 12
-rw-r--r-- 1 elasticsearch elasticsearch  985 Aug 29 08:32 kibana.crt
-rw-r--r-- 1 elasticsearch elasticsearch 1350 Aug 29 08:30 kibana.csr
-rw-r--r-- 1 elasticsearch elasticsearch 1679 Aug 29 08:30 kibana.key

5.4复制容器内证书到宿主机

另开一个窗口，使用docker cp将容器内上述生成的证书拷贝到宿主机

#在宿主机执行下列命令
##下面的证书拷贝到es的certs中，供es使用
docker cp es:/usr/share/elasticsearch/elastic-stack-ca.p12 /home/ec2-user/k8s/elk/es/certs/elastic-stack-ca.p12
docker cp es:/usr/share/elasticsearch/elastic-certificates.p12 /home/ec2-user/k8s/elk/es/certs/elastic-certificates.p12
docker cp es:/usr/share/elasticsearch/elasticsearch/http.p12 /home/ec2-user/k8s/elk/es/certs/http.p12

##下面的证书拷贝到kibana的certs中，供kibana使用
docker cp es:/usr/share/elasticsearch/kibana/elasticsearch-ca.pem /home/ec2-user/k8s/elk/kibana/certs/elasticsearch-ca.pem
docker cp es:/usr/share/elasticsearch/kibana/kibana.crt /home/ec2-user/k8s/elk/kibana/certs/kibana.crt
docker cp es:/usr/share/elasticsearch/kibana/kibana.csr /home/ec2-user/k8s/elk/kibana/certs/kibana.csr
docker cp es:/usr/share/elasticsearch/kibana/kibana.key /home/ec2-user/k8s/elk/kibana/certs/kibana.key

复制完后，可以将docker容器停掉。

六、开始用K8S部署ES集群

首先，看下es目录下的文件

6.1将安全证书添加到Secret中

• elastic-certificates.p12
  • 这个文件是 Elasticsearch 用于节点间加密通信的证书。
• http.p12
  • 这是 Elasticsearch 用于 HTTP 客户端连接安全访问的证书。

##创建命名空间
kubectl create namespace renpho-erp-common
##执行下列命令创建Secret
kubectl create secret generic es-certificates \
  --from-file=/home/ec2-user/k8s/elk/es/certs/elastic-certificates.p12 \
  --from-file=/home/ec2-user/k8s/elk/es/certs/http.p12 \
  -n renpho-erp-common
##查看Secret
kubectl get secret -n renpho-erp-common

6.2运行ES集群

依次执行下列命令

#ES配置文件创建
kubectl apply -f config-map-es.yaml
#ES Service，StatefulSet创建
kubectl apply -f delpoy-es2.yaml
#查看运行状态
kubectl get pod -n renpho-erp-common|grep elastic

浏览器输入：https://192.168.6.220:30092，检查服务是否部署成功

浏览器输入：192.168.6.220:30092/_cat/nodes?v，检查elasticsearch集群是否正常

浏览器输入：https://192.168.6.220:30092/_cluster/state/master_node,nodes?pretty，检查elasticsearch集群详情

到此，使用K8s部署ElasticSearch成功！

部署过程中遇到很多问题，感谢下面博主给出的参考，参考链接

ElasticSearch 8.12.0 K8S部署实践【超详细】【一站式】

Elasticsearch8.11集群部署

k8s集群部署elk