高可用elasticsearch集群搭建

目录

一、环境准备

二、机器配置

   2.1 创建用户

  2.2 修改用户权限

  2.3 解析主机名

  2.4  优化最大文件数

 2.5 优化最大进程数

 2.6 优化虚拟内存

 2.7 重载配置

三、部署

3.1 创建文件夹并赋予权限

3.2 解压安装包并赋予权限

3.3 配置环境变量

3.4 创建数据、证书存放目录并赋予权限

3.5 签发证书

3.6 设置集群多节点 HTTP 证书

3.7 解压证书并分发给其他节点

3.8 配置文件修改配置

3.9 配置文件下发给其他节点并修改

3.10 JVM参数配置

3.11 启动集群

3.12 修改HTTP登录密码

3.13 页面访问验证

3.14 服务关闭

四、安装IK分词器

---

概述

       Elasticsearch（简称为ES）是一个开源的分布式搜索和分析引擎，构建在Apache Lucene搜索引擎库之上。它提供了一个强大的全文搜索和实时分析功能，适用于大规模的数据存储、检索和分析。

Elasticsearch的主要特点和功能：

1. 分布式架构：Elasticsearch采用分布式架构，可以水平扩展到数百台服务器，处理大规模的数据存储和查询。

2. 全文搜索：Elasticsearch提供了高效的全文搜索功能，支持复杂的查询、分词、词组匹配、模糊搜索等。

3. 实时数据：Elasticsearch能够实时索引和分析数据，可以快速地处理大量的实时数据流，适用于日志分析、监控系统等场景。

4. 多种数据类型：除了文本数据，Elasticsearch还支持结构化数据、地理空间数据等多种数据类型的索引和查询。

5. RESTful API：Elasticsearch提供了简单易用的RESTful API，方便开发人员进行数据的索引、查询和管理。

6. 近乎实时的搜索性能：Elasticsearch能够在毫秒级别内返回搜索结果，适用于对搜索性能要求较高的应用场景。

7. 可扩展性：Elasticsearch支持插件机制，可以扩展其功能，满足不同场景下的需求。

8. 安全性：Elasticsearch提供了访问控制、认证和加密等安全功能，保护数据的安全性和隐私性。

一、环境准备

    部署模式：uap的高可用es集群采用三节点的无主模式。

    es版本：使用es版本为 v8.11.0 。

    官网地址：Elasticsearch 平台 — 大规模查找实时答案 | Elastic

    jdk版本：使用es内嵌的jdk21，无需额外安装jdk环境。

IP地址	操作系统	主机名	角色
192.168.122.118	Centos7.6	master.vteamcloud.com	master&data节点
192.168.122.119	Centos7.6	node1.vteamcloud.com	master&data节点
192.168.122.120	Centos7.6	node2.vteamcloud.com	master&data节点

二、机器配置

   2.1 创建用户

     es不能用root用户进行部署，得在每个机器上新建一个用户，部署的步骤都在这个新用户上进行。

# 添加一个用户  elasticsearch ，密码   elasticsearch 
useradd elasticsearch && echo elasticsearch|passwd --stdin elasticsearch

  2.2 修改用户权限

[root@okd ~]# visudo
# 增加一行普通用户权限内容
elasticsearch ALL=(ALL) NOPASSWD:ALL

  2.3 解析主机名

vim /etc/hosts
 
# 添加下面内容
192.168.122.118  master.vteamcloud.com
192.168.122.119  node1.vteamcloud.com
192.168.122.120  node2.vteamcloud.com

  2.4  优化最大文件数

vim /etc/security/limits.conf
 
# 末尾添加下面内容
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 6553

 2.5 优化最大进程数

vim /etc/security/limits.d/20-nproc.conf
 
## 末尾添加下面内容
*          soft    nproc     4096
root       soft    nproc     unlimited

 2.6 优化虚拟内存

vim /etc/sysctl.conf
 
## 添加下面内容
vm.max_map_count=262144

 2.7 重载配置

sysctl -p

三、部署

3.1 创建文件夹并赋予权限

mkdir -p /opt/module/
chown -R elasticsearch.elasticsearch /opt/module/

3.2 解压安装包并赋予权限

tar -xf elasticsearch-8.11.0-linux-x86_64.tar.gz -C /opt/module/
chown -R elasticsearch.elasticsearch /opt/module/elasticsearch-8.11.0

3.3 配置环境变量

vim /etc/profile
 
## 末尾添加下面内容
export JAVA_HOME=/opt/module/elasticsearch-8.11.0/jdk
export ES_HOME=/opt/module/elasticsearch-8.11.0
export PATH=$PATH:$ES_HOME/bin:$JAVA_HOME/bin

# 刷新环境变量
source /etc/profile

3.4 创建数据、证书存放目录并赋予权限

mkdir -p /opt/module/elasticsearch-8.11.0/data
mkdir -p /opt/module/elasticsearch-8.11.0/config/certs
chown -R elasticsearch:elasticsearch /opt/module/elasticsearch-8.11.0

 到这一步为止，三台机器的操作是一模一样的。

3.5 签发证书

# 在第一台服务器节点 master.vteamcloud.com 设置集群多节点通信密钥
# 切换用户
su - elasticsearch
 
cd /opt/module/elasticsearch-8.11.0/bin
 
[elasticsearch@okd bin]$./elasticsearch-certutil ca
 
warning: ignoring JAVA_HOME=/opt/module/elasticsearch-8.11.0/jdk; using bundled JDK
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.
The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.
 
Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority
 
By default the 'ca' mode produces a single PKCS#12 output file which holds:
    * The CA certificate
    * The CAs private key
 
If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key
 
Please enter the desired output file [elastic-stack-ca.p12]: # 回车即可
Enter password for elastic-stack-ca.p12 :  # 回车即可
 
# 用 ca 证书签发节点证书，过程中需按三次回车键,生成目录：es的home:/opt/elasticsearch-8.11.0/
[elasticsearch@okd bin]$ ./elasticsearch-certutil cert --ca elastic-stack-ca.p12
 
If you specify any of the following options:
    * -pem (PEM formatted output)
    * -multiple (generate multiple certificates)
    * -in (generate certificates from an input file)
then the output will be be a zip file containing individual certificate/key files
 
Enter password for CA (elastic-stack-ca.p12) :  # 回车即可
Please enter the desired output file [elastic-certificates.p12]:  # 回车即可
Enter password for elastic-certificates.p12 :  # 回车即可
 
Certificates written to /opt/module/elasticsearch-8.11.0/elastic-certificates.p12
 
This file should be properly secured as it contains the private key for
your instance.
This file is a self contained file and can be copied and used 'as is'
For each Elastic product that you wish to configure, you should copy
this '.p12' file to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.
 
For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.
 
# 将生成的证书文件移动到 config/certs 目录中
[elasticsearch@okd bin]$ cd /opt/module/elasticsearch-8.11.0/
[elasticsearch@okd elasticsearch-8.11.0]$ ls -l | grep "elastic-"
-rw-------  1 elasticsearch elasticsearch   3596 Feb 10 16:05 elastic-certificates.p12
-rw-------  1 elasticsearch elasticsearch   2672 Feb 10 16:03 elastic-stack-ca.p12
[elasticsearch@okd elasticsearch-8.11.0]$
[elasticsearch@okd elasticsearch-8.11.0]$ mv elastic-certificates.p12 config/certs/
[elasticsearch@okd elasticsearch-8.11.0]$ mv elastic-stack-ca.p12 config/certs/

3.6 设置集群多节点 HTTP 证书

# 签发 Https 证书
[elasticsearch@okd elasticsearch-8.11.0]$ cd /opt/module/elasticsearch-8.11.0/bin/
[elasticsearch@okd bin]$ ./elasticsearch-certutil http
warning: ignoring JAVA_HOME=/opt/module/elasticsearch-8.11.0/jdk; using bundled JDK
 
## Elasticsearch HTTP Certificate Utility
The 'http' command guides you through the process of generating certificates
for use on the HTTP (Rest) interface for Elasticsearch.
This tool will ask you a number of questions in order to generate the right
set of files for your needs.
## Do you wish to generate a Certificate Signing Request (CSR)?
A CSR is used when you want your certificate to be created by an existing
Certificate Authority (CA) that you do not control (that is, you do not have
access to the keys for that CA).
If you are in a corporate environment with a central security team, then you
may have an existing Corporate CA that can generate your certificate for you.
Infrastructure within your organisation may already be configured to trust this
CA, so it may be easier for clients to connect to Elasticsearch if you use a
CSR and send that request to the team that controls your CA.
If you choose not to generate a CSR, this tool will generate a new certificate
for you. That certificate will be signed by a CA under your control. This is a
quick and easy way to secure your cluster with TLS, but you will need to
configure all your clients to trust that custom CA.
######################################################
# 是否生成CSR，选择 N ，不需要                           #
######################################################
Generate a CSR? [y/N]N
 
## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?
 
If you have an existing CA certificate and key, then you can use that CA to
sign your new http certificate. This allows you to use the same CA across
multiple Elasticsearch clusters which can make it easier to configure clients,
and may be easier for you to manage.
 
If you do not have an existing CA, one will be generated for you.
######################################################
# 是否使用已经存在的CA证书，选择 y ，因为已经创建签发好了CA    #
######################################################
Use an existing CA? [y/N]y
 
## What is the path to your CA?
Please enter the full pathname to the Certificate Authority that you wish to
use for signing your new http certificate. This can be in PKCS#12 (.p12), JKS
(.jks) or PEM (.crt, .key, .pem) format.
######################################################
# 指定CA证书的路径地址，CA Path:后写绝对路径               #
######################################################
CA Path: /opt/module/elasticsearch-8.11.0/config/certs/elastic-stack-ca.p12
Reading a PKCS12 keystore requires a password.
It is possible for the keystore's password to be blank,
in which case you can simply press <ENTER> at the prompt
 
######################################################
# 设置密钥库的密码，直接 回车 即可                         #
######################################################
Password for elastic-stack-ca.p12:
 
## How long should your certificates be valid?
 
Every certificate has an expiry date. When the expiry date is reached clients
will stop trusting your certificate and TLS connections will fail.
Best practice suggests that you should either:
(a) set this to a short duration (90 - 120 days) and have automatic processes
to generate a new certificate before the old one expires, or
(b) set it to a longer duration (3 - 5 years) and then perform a manual update
a few months before it expires.
 
You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D)
######################################################
# 设置证书的失效时间，这里的y表示年，5y则代表失效时间5年       #
######################################################
For how long should your certificate be valid? [5y] 5y
 
## Do you wish to generate one certificate per node?
 
If you have multiple nodes in your cluster, then you may choose to generate a
separate certificate for each of these nodes. Each certificate will have its
own private key, and will be issued for a specific hostname or IP address.
 
Alternatively, you may wish to generate a single certificate that is valid
across all the hostnames or addresses in your cluster.
 
If all of your nodes will be accessed through a single domain
(e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it
simpler to generate one certificate with a wildcard hostname (*.es.example.com)
and use that across all of your nodes.
 
However, if you do not have a common domain name, and you expect to add
additional nodes to your cluster in the future, then you should generate a
certificate per node so that you can more easily generate new certificates when
you provision new nodes.
 
######################################################
# 是否需要为每个节点都生成证书，选择 N 无需每个节点都配置证书   #
######################################################
Generate a certificate per node? [y/N]N
 
## Which hostnames will be used to connect to your nodes?
These hostnames will be added as "DNS" names in the "Subject Alternative Name"
(SAN) field in your certificate.
You should list every hostname and variant that people will use to connect to
your cluster over http.
Do not list IP addresses here, you will be asked to enter them later.
 
If you wish to use a wildcard certificate (for example *.es.example.com) you
can enter that here.
 
Enter all the hostnames that you need, one per line.
######################################################
# 输入需连接集群节点主机名信息，一行输入一个IP地址，空行回车结束 #
######################################################
When you are done, press <ENTER> once more to move on to the next step.
 
master.vteamcloud.com
node1.vteamcloud.com
node2.vteamcloud.com
 
You entered the following hostnames.
 
 - master.vteamcloud.com
 - node1.vteamcloud.com
 - node2.vteamcloud.com
 
####################################################
# 确认以上是否为正确的配置，输入 Y 表示信息正确            #
####################################################
Is this correct [Y/n]Y
 
## Which IP addresses will be used to connect to your nodes?
If your clients will ever connect to your nodes by numeric IP address, then you
can list these as valid IP "Subject Alternative Name" (SAN) fields in your
certificate.
 
If you do not have fixed IP addresses, or not wish to support direct IP access
to your cluster then you can just press <ENTER> to skip this step.
 
Enter all the IP addresses that you need, one per line.
####################################################
# 输入需连接集群节点IP信息，一行输入一个IP地址，空行回车结束 #
####################################################
When you are done, press <ENTER> once more to move on to the next step.
 
192.168.122.118
192.168.122.119
192.168.122.120
 
 
You entered the following IP addresses.
 
 - 192.168.122.118
 - 192.168.122.119
 - 192.168.122.120
 
####################################################
# 确认以上是否为正确的配置，输入 Y 表示信息正确            #
####################################################
Is this correct [Y/n]Y
 
## Other certificate options
The generated certificate will have the following additional configuration
values. These values have been selected based on a combination of the
information you have provided above and secure defaults. You should not need to
change these values unless you have specific requirements.
 
Key Name: master.vteamcloud.com
Subject DN: CN=master.vteamcloud.com
Key Size: 2048
 
####################################################
# 是否要更改以上这些选项，选择 N ，不更改证书选项配置       #
####################################################
Do you wish to change any of these options? [y/N]N
 
## What password do you want for your private key(s)?
 
Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12".
This type of keystore is always password protected, but it is possible to use a
blank password.
 
####################################################
# 是否要给证书加密，不需要加密，两次 回车 即可             #
####################################################
If you wish to use a blank password, simply press <enter> at the prompt below.
Provide a password for the "http.p12" file:  [<ENTER> for none]
 
## Where should we save the generated files?
A number of files will be generated including your private key(s),
public certificate(s), and sample configuration options for Elastic Stack products.
These files will be included in a single zip archive.
What filename should be used for the output zip file? [/opt/module/elasticsearch-8.11.0/elasticsearch-ssl-http.zip]
Zip file written to /opt/module/elasticsearch-8.11.0/elasticsearch-ssl-http.zip

3.7 解压证书并分发给其他节点

# 解压
[elasticsearch@okd bin]$ cd /opt/module/elasticsearch-8.11.0/
[elasticsearch@okd elasticsearch-8.11.0]$ unzip elasticsearch-ssl-http.zip
# 移动证书
[elasticsearch@okd elasticsearch-8.11.0]$ mv ./elasticsearch/http.p12 config/certs/
[elasticsearch@okd elasticsearch-8.11.0]$ mv ./kibana/elasticsearch-ca.pem config/certs/
 
# 将证书分发到其他节点02 03
[elasticsearch@okd elasticsearch-8.11.0]$ cd /opt/module/elasticsearch-8.11.0/config/certs
[elasticsearch@okd certs]$ ll
total 16
-rw------- 1 elasticsearch elasticsearch 3596 Feb 10 16:05 elastic-certificates.p12
-rw-rw-r-- 1 elasticsearch elasticsearch 1200 Feb 10 16:13 elasticsearch-ca.pem
-rw------- 1 elasticsearch elasticsearch 2672 Feb 10 16:03 elastic-stack-ca.p12
-rw-rw-r-- 1 elasticsearch elasticsearch 3652 Feb 10 16:13 http.p12
[elasticsearch@okd certs]$ scp * node1.vteamcloud.com:/opt/module/elasticsearch-8.11.0/config/certs/
[elasticsearch@okd certs]$ scp * node2.vteamcloud.com:/opt/module/elasticsearch-8.11.0/config/certs/

3.8 配置文件修改配置

[elasticsearch@okd certs]$ cd /opt/module/elasticsearch-8.11.0/config/
[elasticsearch@okd config]$ vim elasticsearch.yml
cluster.name: uap-es
node.name: es-master.vteamcloud.com
path.data: /opt/module/elasticsearch-8.11.0/data
path.logs: /opt/module/elasticsearch-8.11.0/logs
network.host: 0.0.0.0
http.port: 9200
# 种子主机，在选举时用于发现其他主机的，最好配置多个
discovery.seed_hosts: ["master.vteamcloud.com","node1.vteamcloud.com","node2.vteamcloud.com"]
cluster.initial_master_nodes: ["es-master.vteamcloud.com", "es-node1.vteamcloud.com","es-node2.vteamcloud.com"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
 enabled: true
 keystore.path: /opt/module/elasticsearch-8.11.0/config/certs/http.p12
 keystore.password: 123456  #如果生成证书时设置了密码则要添加密码配置
 truststore.path: /opt/module/elasticsearch-8.11.0/config/certs/http.p12
 truststore.password: 123456 #如果生成证书时设置了密码则要添加密码配置
xpack.security.transport.ssl:
 enabled: true
 verification_mode: certificate
 keystore.path: /opt/module/elasticsearch-8.11.0/config/certs/elastic-certificates.p12
 keystore.password: 123456  #如果生成证书时设置了密码则要添加密码配置
 truststore.path: /opt/module/elasticsearch-8.11.0/config/certs/elastic-certificates.p12
 truststore.password: 123456 #如果生成证书时设置了密码则要添加密码配置
http.host: [_local_, _site_]
ingest.geoip.downloader.enabled: false
xpack.security.http.ssl.client_authentication: none

【注意】：

1. xpack.security.http.ssl & xpack.security.transport.ssl后的子配置需要空一格，遵循yml的格式要求
2. 如果不需要后续的http证书认证或者用户密码认证可以将xpack.security相关的功能falase关闭掉

xpack.security.http.ssl:
 enabled: false
xpack.security.transport.ssl:
 enabled: false

有些业务使用场景中，可能会遇到跨域问题，当elasticsearch需要涉及到跨域问题时，可以在配置文件中最后增加配置：

http.cors.enabled: true
http.cors.allow-origin: "*"

3.9 配置文件下发给其他节点并修改

[elasticsearch@okd elasticsearch-8.11.0]$ scp config/elasticsearch.yml node1.vteamcloud.com:/opt/module/elasticsearch-8.11.0/config/
[elasticsearch@okd elasticsearch-8.11.0]$ scp config/elasticsearch.yml node2.vteamcloud.com:/opt/module/elasticsearch-8.11.0/config/
 
# node1修改 config/elasticsearch.yml
[elasticsearch@node1 ~]# vim /opt/module/elasticsearch-8.11.0/config/elasticsearch.yml
# 设置节点名称
node.name: es-node1.vteamcloud.com
 
# node2修改 config/elasticsearch.yml
[elasticsearch@node2 ~]# vim /opt/module/elasticsearch-8.11.0/config/elasticsearch.yml
# 设置节点名称
node.name: es-node2.vteamcloud.com

3.10 JVM参数配置

    es的本质是一个java服务，也需要jvm参数。es的jvm参数在config文件夹下的jvm.options文件中，修改此文件配置jvm参数即可。

vim jvm.options
 
# 配置内存参数
-Xms2g
-Xmx2g

3.11 启动集群

每台节点依次启动（无顺序要求，只要多于2台，就可以启动集群，这就是es的无主模式，自动识别集群，选举master）：

[elasticsearch@okd elasticsearch-8.11.0]$ /opt/module/elasticsearch-8.11.0/bin/elasticsearch  -d
[elasticsearch@es02 elasticsearch-8.11.0]$ /opt/module/elasticsearch-8.11.0/bin/elasticsearch  -d
[elasticsearch@es03 elasticsearch-8.11.0]$ /opt/module/elasticsearch-8.11.0/bin/elasticsearch  -d

3.12 修改HTTP登录密码

# 手工指定elastic的新密码 (-i参数)
[elasticsearch@okd ~]$ /opt/module/elasticsearch-8.11.0/bin/elasticsearch-reset-password -u elastic -i
warning: ignoring JAVA_HOME=/opt/module/elasticsearch-8.11.0/jdk; using bundled JDK
bThis tool will reset the password of the [elastic] user.
You will be prompted to enter the password.
Please confirm that you would like to continue [y/N]y
Did not understand answer 'by'
Please confirm that you would like to continue [y/N]y
 
 
Enter password for [elastic]: # 输入用户elastic的密码
Re-enter password for [elastic]: # 输入用户elastic的密码
Password for the [elastic] user successfully reset.

3.13 页面访问验证

https://ip:9200 (注意是https)

账号密码为上面创建的：elastic / elastic的密码

3.14 服务关闭

[elasticsearch@okd ~]$ ps -ef | grep elasticsearch|grep -vE "grep|controller" |awk -F" " '{print $2}' | xargs kill -9

四、安装IK分词器

  将附件中ik分词器安装包放到es的plugins/ik/目录下，解压后重启es服务即可。ik目录需自己创建。

mkdir ik
unzip elasticsearch-analysis-ik-8.1.0.zip

五、可能会遇到的问题

1.  集群节点相互连不上

   解决方案：检查防火墙是否开启

2.  报错：不能同时使用两个key

   解决方案：

# 检查是否有缓存秘钥
  ./elasticsearch-keystore  list

# 删除缓存的秘钥
./elasticsearch-keystore  remove ....