Elasticsearch集群Linux搭建

目录

1、前期准备

2、集群搭建

       1)从官网下载ES 安装包
       2)新增ES用户
       3)上传解压ES
       4)配置ES
       5)启动前提前期解决的问题
       6)启动、关闭ES
       7)集群验证
       8)安装插件(kibana-5.4.0-linux-x86_64)
       9)集群验证

3、集群搭建出现的问题

       问题1:Maven有问题包下载不下来
       问题2:配置错误
       问题3:修改虚拟机及Linux环境
       问题4:max file descriptors
       问题5:vm.max_map_count
       问题6:内存不足
       问题7:Eclipse源码编译打包步骤

————————————————————————————-

1、前期准备

下载地址:
Java程序员软件包
http://blog.csdn.net/silentwolfyh/article/details/50936377
大数据程序员软件包
http://blog.csdn.net/silentwolfyh/article/details/50936459

Java版本:java version “1.8.0_77”
系统版本:Centos 6.7
机器节点:hadoop11,hadoop12,hadoop13
软件:
elasticsearch-5.4.0.tar.gz
elasticsearch-analysis-ik-5.4.0.zip
kibana-5.4.0-linux-x86_64.tar.gz

2、集群搭建

       1)从官网下载ES 安装包

elasticsearch-5.4.0.tar.gz

       2)新增ES用户

注意: Linux新增,用户用户名为es,密码为hadoop

[root@hadoop11 ~]# adduser es
[root@hadoop11 ~]# passwd es
更改用户 es 的密码 。
新的 密码:
无效的密码: 它基于字典单词
无效的密码: 过于简单
重新输入新的 密码:
passwd: 所有的身份验证令牌已经成功更新。
[root@hadoop11 ~]# su es
[es@hadoop11 root]$

       3)上传解压ES

注意:一定要切换用户,不能用root用户解压,不能用root用户启动,ES官网规定的。

[es@hadoop11 root] cd/usr/app/[es@hadoop11 ] pwd
/home/es
[es@hadoop11 es]$ tar -zxvf elasticsearch-5.4.0.tar.gz

       4)配置ES

注意:第3个节点都要修改node.name
编辑文档如下:
[es@hadoop11 root]$ vi /home/es/elasticsearch-5.4.0/config/elasticsearch.yml
文档配置解说

cluster.name: es-cluster #集群名称
node.name: es-hadoop11 #节点名称
path.data: /home/es/elasticsearch-5.4.0/data #数据文件存储路径
path.logs: /home/es/elasticsearch-5.4.0/logs #log文件存储路径
network.host: 192.168.200.11 #默认网络连接地址
http.port: 9200 #连接端口
discovery.zen.minimum_master_nodes: 2 #这个参数来保证集群中的节点可以知道其它N个有master资格的节点。默认为1,对于大的集群来说,可以设置大一点的值(2-4)
discovery.zen.ping.unicast.hosts: [“192.168.7.126”, “192.168.7.125”] #集群中master节点的初始列表,可以通过这些节点来自动发现新加入集群的节点。
discovery.zen.ping_timeout: 120s
bootstrap.system_call_filter: false
http.cors.enabled: true
http.cors.allow-origin: “*”

文档主要不同的配置
Hadoop11

cluster.name: es-cluster
node.name: es-hadoop11
path.data: /home/es/elasticsearch-5.4.0/data
path.logs: /home/es/elasticsearch-5.4.0/logs
network.host: 192.168.200.11
http.port: 9200
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["192.168.200.11", "192.168.200.12", "192.168.200.13"]
discovery.zen.ping_timeout: 120s
bootstrap.system_call_filter: false
http.cors.enabled: true
http.cors.allow-origin: "*"

Hadoop12

cluster.name: es-cluster
node.name: es-hadoop12
path.data: /home/es/elasticsearch-5.4.0/data
path.logs: /home/es/elasticsearch-5.4.0/logs
network.host: 192.168.200.12
http.port: 9200
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["192.168.200.11", "192.168.200.12", "192.168.200.13"]
discovery.zen.ping_timeout: 120s
bootstrap.system_call_filter: false
http.cors.enabled: true
http.cors.allow-origin: "*"

Hadoop13

cluster.name: es-cluster
node.name: es-hadoop13
path.data: /home/es/elasticsearch-5.4.0/data
path.logs: /home/es/elasticsearch-5.4.0/logs
network.host: 192.168.200.13
http.port: 9200
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["192.168.200.11", "192.168.200.12", "192.168.200.13"]
discovery.zen.ping_timeout: 120s
bootstrap.system_call_filter: false
http.cors.enabled: true
http.cors.allow-origin: "*"
       5)启动前提前期解决的问题

问题:max file descriptors
问题:vm.max_map_count
问题:内存不足

       6)启动、关闭ES
/home/es/elasticsearch-5.4.0/bin/elasticsearch    ---前台启动
/home/es/elasticsearch-5.4.0/bin/elasticsearch   -d  ---后台启动
查看启动进程:jps | grep Elasticsearch
关闭:
kill -15  pid
       7)集群验证

http://hadoop11:9200/
这里写图片描述

       8)安装插件(kibana-5.4.0-linux-x86_64)

【注意】用es用户解压,修改kibana-5.4.0-linux-x86_64/config/kibana.yml
修改配置

server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.200.11:9200"

执行 bin/kibana 即可
注意:
1. 需要将server.host修改一下,否则远程无法访问,这个配置在Kibana4.6.1(对应ES2.4.0)的时候还不需要配置,但是5.4.0就需要了
2. 之前ES和Kibana配合还需要使用plugin安装一些Marvel,sense等,现在都不需要了,DevTools就是之前的Sense

http://192.168.200.11:5601/
这里写图片描述

       9)集群验证

GET /_cat/health?v

epoch      timestamp cluster    status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1500041888 22:18:08  es-cluster green           3         3      2   1    0    0        0             0                  -                100.0%

如何快速了解集群的健康状况?green、yellow、red?

green:每个索引的primary shard和replica shard都是active状态的
yellow:每个索引的primary shard都是active状态的,但是部分replica shard不是active状态,处于不可用的状态
red:不是所有索引的primary shard都是active状态的,部分索引有数据丢失了

为什么现在会处于一个yellow状态?
我们现在就一个笔记本电脑,就启动了一个es进程,相当于就只有一个node。现在es中有一个index,就是kibana自己内置建立的index。由于默认的配置是给每个index分配5个primary shard和5个replica shard,而且primary shard和replica shard不能在同一台机器上(为了容错)。现在kibana自己建立的index是1个primary shard和1个replica shard。当前就一个node,所以只有1个primary shard被分配了和启动了,但是一个replica shard没有第二台机器去启动。

做一个小实验:此时只要启动第二个es进程,就会在es集群中有2个node,然后那1个replica shard就会自动分配过去,然后cluster status就会变成green状态。

3、集群搭建出现的问题

问题1:Maven有问题包下载不下来

点击项目 —> 右键 —> maven —> Update Project

问题2:配置错误

https://github.com/elastic/elasticsearch/issues/22899
bootstrap.system_call_filter: true

[es@hadoop13 root]$ /home/es/elasticsearch-5.4.0/bin/elasticsearch
[2017-07-13T23:39:55,695][WARN ][o.e.b.JNANatives         ] unable to install syscall filter: 
java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
    at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:363) ~[elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:638) ~[elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:215) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:99) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.0.jar:5.4.0]
[2017-07-13T23:39:55,805][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=无法分配内存
[2017-07-13T23:39:55,806][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2017-07-13T23:39:55,807][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2017-07-13T23:39:55,807][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example: 
    # allow user 'es' mlockall
    es soft memlock unlimited
    es hard memlock unlimited
[2017-07-13T23:39:55,807][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2017-07-13T23:39:59,020][INFO ][o.e.n.Node               ] [es-hadoop13] initializing ...
[2017-07-13T23:40:00,737][INFO ][o.e.e.NodeEnvironment    ] [es-hadoop13] using [1] data paths, mounts [[/ (/dev/sda3)]], net usable_space [9.3gb], net total_space [15.4gb], spins? [possibly], types [ext4]
[2017-07-13T23:40:00,737][INFO ][o.e.e.NodeEnvironment    ] [es-hadoop13] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-07-13T23:40:00,739][INFO ][o.e.n.Node               ] [es-hadoop13] node name [es-hadoop13], node ID [6ga7KPpTQmiQtTlxo4plQQ]
[2017-07-13T23:40:00,739][INFO ][o.e.n.Node               ] [es-hadoop13] version[5.4.0], pid[2653], build[780f8c4/2017-04-28T17:43:27.229Z], OS[Linux/2.6.32-573.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_77/25.77-b03]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [aggs-matrix-stats]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [ingest-common]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [lang-expression]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [lang-groovy]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [lang-mustache]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [lang-painless]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [percolator]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [reindex]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [transport-netty3]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [transport-netty4]
[2017-07-13T23:40:14,647][INFO ][o.e.p.PluginsService     ] [es-hadoop13] no plugins loaded
[2017-07-13T23:40:33,836][INFO ][o.e.d.DiscoveryModule    ] [es-hadoop13] using discovery type [zen]
[2017-07-13T23:40:36,207][INFO ][o.e.n.Node               ] [es-hadoop13] initialized
[2017-07-13T23:40:36,207][INFO ][o.e.n.Node               ] [es-hadoop13] starting ...
[2017-07-13T23:40:37,699][INFO ][o.e.t.TransportService   ] [es-hadoop13] publish_address {192.168.200.13:9300}, bound_addresses {192.168.200.13:9300}
[2017-07-13T23:40:37,709][INFO ][o.e.b.BootstrapChecks    ] [es-hadoop13] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
memory locking requested for elasticsearch process but memory is not locked
max number of threads [1024] for user [es] is too low, increase to at least [2048]
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-07-13T23:40:37,763][INFO ][o.e.n.Node               ] [es-hadoop13] stopping ...
[2017-07-13T23:40:37,830][INFO ][o.e.n.Node               ] [es-hadoop13] stopped
[2017-07-13T23:40:37,831][INFO ][o.e.n.Node               ] [es-hadoop13] closing ...
[2017-07-13T23:40:37,878][INFO ][o.e.n.Node               ] [es-hadoop13] closed
[es@hadoop13 root]$ vi /home/es/elasticsearch-5.4.0/config/elasticsearch.yml
[es@hadoop13 root]$
问题3:修改虚拟机及Linux环境
[es@hadoop12 config]$ /home/es/elasticsearch-5.4.0/bin/elasticsearch 
[2017-07-14T17:04:12,114][INFO ][o.e.n.Node               ] [es-hadoop12] initializing ...
[2017-07-14T17:04:13,332][INFO ][o.e.e.NodeEnvironment    ] [es-hadoop12] using [1] data paths, mounts [[/ (/dev/sda3)]], net usable_space [9.2gb], net total_space [15.4gb], spins? [possibly], types [ext4]
[2017-07-14T17:04:13,332][INFO ][o.e.e.NodeEnvironment    ] [es-hadoop12] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-07-14T17:04:13,333][INFO ][o.e.n.Node               ] [es-hadoop12] node name [es-hadoop12], node ID [laYcbaV_QLKGjW7uKaP5BQ]
[2017-07-14T17:04:13,334][INFO ][o.e.n.Node               ] [es-hadoop12] version[5.4.0], pid[7774], build[780f8c4/2017-04-28T17:43:27.229Z], OS[Linux/2.6.32-573.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_77/25.77-b03]
[2017-07-14T17:04:25,712][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [aggs-matrix-stats]
[2017-07-14T17:04:25,712][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [ingest-common]
[2017-07-14T17:04:25,712][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [lang-expression]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [lang-groovy]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [lang-mustache]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [lang-painless]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [percolator]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [reindex]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [transport-netty3]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [transport-netty4]
[2017-07-14T17:04:25,715][INFO ][o.e.p.PluginsService     ] [es-hadoop12] no plugins loaded
[2017-07-14T17:04:39,353][INFO ][o.e.d.DiscoveryModule    ] [es-hadoop12] using discovery type [zen]
[2017-07-14T17:04:42,769][INFO ][o.e.n.Node               ] [es-hadoop12] initialized
[2017-07-14T17:04:42,770][INFO ][o.e.n.Node               ] [es-hadoop12] starting ...
[2017-07-14T17:04:44,195][INFO ][o.e.t.TransportService   ] [es-hadoop12] publish_address {192.168.200.12:9300}, bound_addresses {192.168.200.12:9300}
[2017-07-14T17:04:44,291][INFO ][o.e.b.BootstrapChecks    ] [es-hadoop12] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
max number of threads [1024] for user [es] is too low, increase to at least [2048]
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2017-07-14T17:04:44,358][INFO ][o.e.n.Node               ] [es-hadoop12] stopping ...
[2017-07-14T17:04:44,606][INFO ][o.e.n.Node               ] [es-hadoop12] stopped
[2017-07-14T17:04:44,606][INFO ][o.e.n.Node               ] [es-hadoop12] closing ...
[2017-07-14T17:04:44,682][INFO ][o.e.n.Node               ] [es-hadoop12] closed
问题4:max file descriptors

错误 max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
在linux系统:/etc/security/limits.conf
编辑该文件,后面加上:

xq soft nofile 65536
xq hard nofile 65536

备注:xq为用户名,同时确保xq用户拥有次命令权限

问题5:vm.max_map_count

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决:切换到root用户修改配置sysctl.conf

vi /etc/sysctl.conf 

添加下面配置:

vm.max_map_count=655360

并执行命令:

sysctl -p

然后,重新启动elasticsearch,即可启动成功。
或者:
是以为操作系统的vm.max_map_count参数设置太小导致的,请使用root用户登录系统,执行以下命令:

sysctl -w vm.max_map_count=655360

并用以下命令查看是否修改成功

sysctl -a | grep "vm.max_map_count"

如果能正常输出655360,则说明修改成功,然后再次启动elasticsearch
把配置好的安装包拷贝一份到其他两台机器上,修改 config/elasticsearch.yml下的node.name和network.host为对于的机器即可。

问题6:内存不足

max number of threads [1024] for user [es] likely too low, increase to at least [2048]
解决:切换到root用户,进入limits.d目录下修改配置文件。

vi /etc/security/limits.d/90-nproc.conf 

修改如下内容:

* soft nproc 1024

修改为

* soft nproc 2048

这里写图片描述
这里写图片描述

修改如下:

-Xms1g
-Xmx1g
问题7:Eclipse源码编译打包步骤

Run as ==> Maven clean
Run as ==> Maven build

  • 1
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 1
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

北京小辉

你的鼓舞将是我最大的动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值