Elasticsearch7.6.2 rpm集群部署及异常处理

本次集群部署的目标：部署2个节点的es集群。

在部署过程中遇到了几个问题：

1. 脑裂问题：由于只有2个节点，未配置node.master参数，导致选举失败，2个节点都是主节点。解决方法配置一个为true，一个为false
2. 集群名称设置问题：开始使用默认集群名称，直接启动起来后，发现两个集群名称不同，修改后再次启动一直报错，最终发现是需要先清除新加节点path.data刚才产生的不一致的数据，再次重启，OK

如有遇到相同问题的，可以参考下下面的解决方案！

基础配置：

1. 系统：CentOS Linux release 7.4
2. Java：openjdk version "1.8.0_262"
3. elasticsearch：7.6.2

镜像地址：

​ https://mirrors.huaweicloud.com/

一、安装(rpm)

[test@testv ~] cd /usr/local/src
#下载
[test@testv ~] wget https://mirrors.huaweicloud.com/elasticsearch/7.6.2/elasticsearch-7.6.2-x86_64.rpm
#安装
[test@testv ~] rpm -ivh elasticsearch-7.6.2-x86_64.rpm
#启动
[test@testv ~] systemctl start elasticsearch.service
#此时启动成功，即可查看信息
[test@testv ~] curl http://192.168.1.55:9200/
{
  "name" : "node-1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "S-iXD_HTsdfe0xWa4YaM4Q",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

二、参数配置

到目前为止，一个节点已经配置完成了，并且启动成功，下面查看下参数的配置情况，由于是使用rpm直接安装的，elasticsearch.yml文件位置在/etc/elasticsearch/elasticsearch.yml

配置路径：/etc/elasticsearch/elasticsearch.yml

其中主要关注的几个参数：

cluster.name: elasticsearch
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.1.55 #也可以为:0.0.0.0
cluster.initial_master_nodes: ["node-1"]

三、2节点集群参数配置

node-1

#集群
cluster.name: elasticsearch
#节点
node.name: node-1
node.master: true
#路径
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
#网络
network.host: 0.0.0.0
#发现
discovery.zen.ping.unicast.hosts: ["192.168.1.55", "192.168.1.56"]
discovery.zen.minimum_master_nodes: 1
cluster.initial_master_nodes: ["node-1"]

node-2

#集群
cluster.name: elasticsearch
#节点
node.name: node-2
node.master: false
#路径
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
#网络
network.host: 192.168.1.56 #也可以为:0.0.0.0
#发现
discovery.zen.ping.unicast.hosts: ["192.168.1.55", "192.168.1.56"]
discovery.zen.minimum_master_nodes: 1
cluster.initial_master_nodes: ["node-1"]

由于本次配置的集群是2个结点，遇到的问题汇总下：

1. cluster.name：集群名称必须相同（下面有一个集群名称不同时的日志信息，可供参考）
2. node.master，必须为一个是true，另一个是false，之前在网上看挺多配置，都是这样，但是不明白原因，在配置中发现，如果不设置或都设置成true，会出现脑裂发问，可能是选举主节点时，只有2个节点，投票相同的原因，都成了主结点

ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.1.55           22          81   0    0.00    0.10     0.12 dilm      *      node-1

ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.1.56           22          81   0    0.00    0.10     0.12 dilm      *      node-1

四、其它问题

1、部署时由于集群名称不同，查看到的日志提示信息

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-wvKQmGjO-1608169008731)(C:\Users\wangshunguang\AppData\Roaming\Typora\typora-user-images\image-20201216101135967.png)]

[2020-12-15T20:02:10,318][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-security]
[2020-12-15T20:02:10,318][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-sql]
[2020-12-15T20:02:10,318][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-voting-only-node]
[2020-12-15T20:02:10,319][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-watcher]
[2020-12-15T20:02:10,319][INFO ][o.e.p.PluginsService     ] [node-1] no plugins loaded
[2020-12-15T20:02:14,779][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-12-15T20:02:15,408][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/31373] [Main.cc@110] controller (64 bit): Version 7.6.2 (Build e06ef9d86d5332) Copyright (c) 2020 Elasticsearch BV
[2020-12-15T20:02:16,190][DEBUG][o.e.a.ActionModule       ] [node-1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-12-15T20:02:16,360][INFO ][o.e.d.DiscoveryModule    ] [node-1] using discovery type [zen] and seed hosts providers [settings]
[2020-12-15T20:02:17,464][INFO ][o.e.n.Node               ] [node-1] initialized
[2020-12-15T20:02:17,465][INFO ][o.e.n.Node               ] [node-1] starting ...
[2020-12-15T20:02:17,632][INFO ][o.e.t.TransportService   ] [node-1] publish_address {192.168.1.55:9300}, bound_addresses {192.168.1.55:9300}
[2020-12-15T20:02:18,151][INFO ][o.e.b.BootstrapChecks    ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-12-15T20:02:18,180][INFO ][o.e.c.c.Coordinator      ] [node-1] cluster UUID [Gx-iXD_HTAqe0xWa4YaM4Q]
[2020-12-15T20:02:18,307][WARN ][o.e.d.HandshakingTransportAddressConnector] [node-1] handshake failed for [connectToRemoteMasterNode[192.168.1.56:9300]]
java.lang.IllegalStateException: handshake with [{192.168.1.56:9300}{lzdm4iFJRHmBmfF8-ShuBQ}{192.168.1.56}{192.168.1.56:9300}] failed: remote cluster name [test_es] does not match local cluster name [elasticsearch]

2、节点无法加入集群，报错

​ 原因：集群名称不同，变更后，无法加入集群

​ 解决：清除的数据为：path.data

Caused by: java.lang.IllegalStateException: failure when sending a validation request to node

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-KAzEuvsd-1608169008734)(C:\Users\wangshunguang\AppData\Roaming\Typora\typora-user-images\image-20201216093603089.png)]

[2020-12-15T20:53:12,658][INFO ][o.e.c.c.JoinHelper       ] [node-2] failed to join {node-1}{c84s_FXeTJa61n3toqXRcQ}{J7f57VGISO-DxQ3Iy92UPg}{192.168.1.56}{192.168.1.56:9300}{dilm}{ml.machine_memory=16656175104, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={node-2}{-6V2-cTdSZqR5r633Zo44Q}{Ait1Bxx8QWuHD0fGZsfQfg}{192.168.1.56}{192.168.1.56:9300}{dil}{ml.machine_memory=16656175104, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [node-1][192.168.1.56:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
	at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:514) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1130) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1130) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:244) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) ~[elasticsearch-7.6.2.jar:7.6.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-2][192.168.1.56:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid Gx-iXD_HTAqe0xWa4YaM4Q than local cluster uuid ekMp4oVNStWEqrTjoSNm8Q, rejecting
	at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.2.jar:7.6.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:830) ~[?:?]