Elasticsearch集群Linux搭建

276人阅读 评论(0) 收藏 举报
分类:

目录

1、前期准备

2、集群搭建

       1)从官网下载ES 安装包

       2)新增ES用户

       3)上传解压ES

       4)配置ES

       5)启动前提前期解决的问题

       6)启动、关闭ES

       7)集群验证

       8)安装插件(kibana-5.4.0-linux-x86_64)

       9)集群验证

3、集群搭建出现的问题

       问题1:Maven有问题包下载不下来

       问题2:配置错误

       问题3:修改虚拟机及Linux环境

       问题4:max file descriptors

       问题5:vm.max_map_count

       问题6:内存不足

       问题7:Eclipse源码编译打包步骤

————————————————————————————-

1、前期准备

下载地址:
Java程序员软件包
http://blog.csdn.net/silentwolfyh/article/details/50936377
大数据程序员软件包
http://blog.csdn.net/silentwolfyh/article/details/50936459

Java版本:java version “1.8.0_77”
系统版本:Centos 6.7
机器节点:hadoop11,hadoop12,hadoop13
软件:
elasticsearch-5.4.0.tar.gz
elasticsearch-analysis-ik-5.4.0.zip
kibana-5.4.0-linux-x86_64.tar.gz

2、集群搭建

       1)从官网下载ES 安装包

elasticsearch-5.4.0.tar.gz

       2)新增ES用户

注意: Linux新增,用户用户名为es,密码为hadoop

[root@hadoop11 ~]# adduser es
[root@hadoop11 ~]# passwd es
更改用户 es 的密码 。
新的 密码:
无效的密码: 它基于字典单词
无效的密码: 过于简单
重新输入新的 密码:
passwd: 所有的身份验证令牌已经成功更新。
[root@hadoop11 ~]# su es
[es@hadoop11 root]$

       3)上传解压ES

注意:一定要切换用户,不能用root用户解压,不能用root用户启动,ES官网规定的。

[es@hadoop11 root]cd/usr/app/[es@hadoop11 ] pwd
/home/es
[es@hadoop11 es]$ tar -zxvf elasticsearch-5.4.0.tar.gz

       4)配置ES

注意:第3个节点都要修改node.name
编辑文档如下:
[es@hadoop11 root]$ vi /home/es/elasticsearch-5.4.0/config/elasticsearch.yml
文档配置解说

cluster.name: es-cluster #集群名称
node.name: es-hadoop11 #节点名称
path.data: /home/es/elasticsearch-5.4.0/data #数据文件存储路径
path.logs: /home/es/elasticsearch-5.4.0/logs #log文件存储路径
network.host: 192.168.200.11 #默认网络连接地址
http.port: 9200 #连接端口
discovery.zen.minimum_master_nodes: 2 #这个参数来保证集群中的节点可以知道其它N个有master资格的节点。默认为1,对于大的集群来说,可以设置大一点的值(2-4)
discovery.zen.ping.unicast.hosts: [“192.168.7.126”, “192.168.7.125”] #集群中master节点的初始列表,可以通过这些节点来自动发现新加入集群的节点。
discovery.zen.ping_timeout: 120s
bootstrap.system_call_filter: false
http.cors.enabled: true
http.cors.allow-origin: “*”

文档主要不同的配置
Hadoop11

cluster.name: es-cluster
node.name: es-hadoop11
path.data: /home/es/elasticsearch-5.4.0/data
path.logs: /home/es/elasticsearch-5.4.0/logs
network.host: 192.168.200.11
http.port: 9200
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["192.168.200.11", "192.168.200.12", "192.168.200.13"]
discovery.zen.ping_timeout: 120s
bootstrap.system_call_filter: false
http.cors.enabled: true
http.cors.allow-origin: "*"

Hadoop12

cluster.name: es-cluster
node.name: es-hadoop12
path.data: /home/es/elasticsearch-5.4.0/data
path.logs: /home/es/elasticsearch-5.4.0/logs
network.host: 192.168.200.12
http.port: 9200
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["192.168.200.11", "192.168.200.12", "192.168.200.13"]
discovery.zen.ping_timeout: 120s
bootstrap.system_call_filter: false
http.cors.enabled: true
http.cors.allow-origin: "*"

Hadoop13

cluster.name: es-cluster
node.name: es-hadoop13
path.data: /home/es/elasticsearch-5.4.0/data
path.logs: /home/es/elasticsearch-5.4.0/logs
network.host: 192.168.200.13
http.port: 9200
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["192.168.200.11", "192.168.200.12", "192.168.200.13"]
discovery.zen.ping_timeout: 120s
bootstrap.system_call_filter: false
http.cors.enabled: true
http.cors.allow-origin: "*"

       5)启动前提前期解决的问题

问题:max file descriptors
问题:vm.max_map_count
问题:内存不足

       6)启动、关闭ES

/home/es/elasticsearch-5.4.0/bin/elasticsearch    ---前台启动
/home/es/elasticsearch-5.4.0/bin/elasticsearch   -d  ---后台启动
查看启动进程:jps | grep Elasticsearch
关闭:
kill -15  pid

       7)集群验证

http://hadoop11:9200/
这里写图片描述

       8)安装插件(kibana-5.4.0-linux-x86_64)

【注意】用es用户解压,修改kibana-5.4.0-linux-x86_64/config/kibana.yml
修改配置

server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.200.11:9200"

执行 bin/kibana 即可
注意:
1. 需要将server.host修改一下,否则远程无法访问,这个配置在Kibana4.6.1(对应ES2.4.0)的时候还不需要配置,但是5.4.0就需要了
2. 之前ES和Kibana配合还需要使用plugin安装一些Marvel,sense等,现在都不需要了,DevTools就是之前的Sense

http://192.168.200.11:5601/
这里写图片描述

       9)集群验证

GET /_cat/health?v

epoch      timestamp cluster    status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1500041888 22:18:08  es-cluster green           3         3      2   1    0    0        0             0                  -                100.0%

如何快速了解集群的健康状况?green、yellow、red?

green:每个索引的primary shard和replica shard都是active状态的
yellow:每个索引的primary shard都是active状态的,但是部分replica shard不是active状态,处于不可用的状态
red:不是所有索引的primary shard都是active状态的,部分索引有数据丢失了

为什么现在会处于一个yellow状态?
我们现在就一个笔记本电脑,就启动了一个es进程,相当于就只有一个node。现在es中有一个index,就是kibana自己内置建立的index。由于默认的配置是给每个index分配5个primary shard和5个replica shard,而且primary shard和replica shard不能在同一台机器上(为了容错)。现在kibana自己建立的index是1个primary shard和1个replica shard。当前就一个node,所以只有1个primary shard被分配了和启动了,但是一个replica shard没有第二台机器去启动。

做一个小实验:此时只要启动第二个es进程,就会在es集群中有2个node,然后那1个replica shard就会自动分配过去,然后cluster status就会变成green状态。

3、集群搭建出现的问题

问题1:Maven有问题包下载不下来

点击项目 —> 右键 —> maven —> Update Project

问题2:配置错误

https://github.com/elastic/elasticsearch/issues/22899
bootstrap.system_call_filter: true

[es@hadoop13 root]$ /home/es/elasticsearch-5.4.0/bin/elasticsearch
[2017-07-13T23:39:55,695][WARN ][o.e.b.JNANatives         ] unable to install syscall filter: 
java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
    at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:363) ~[elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:638) ~[elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:215) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:99) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.0.jar:5.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.0.jar:5.4.0]
[2017-07-13T23:39:55,805][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=无法分配内存
[2017-07-13T23:39:55,806][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2017-07-13T23:39:55,807][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2017-07-13T23:39:55,807][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example: 
    # allow user 'es' mlockall
    es soft memlock unlimited
    es hard memlock unlimited
[2017-07-13T23:39:55,807][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2017-07-13T23:39:59,020][INFO ][o.e.n.Node               ] [es-hadoop13] initializing ...
[2017-07-13T23:40:00,737][INFO ][o.e.e.NodeEnvironment    ] [es-hadoop13] using [1] data paths, mounts [[/ (/dev/sda3)]], net usable_space [9.3gb], net total_space [15.4gb], spins? [possibly], types [ext4]
[2017-07-13T23:40:00,737][INFO ][o.e.e.NodeEnvironment    ] [es-hadoop13] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-07-13T23:40:00,739][INFO ][o.e.n.Node               ] [es-hadoop13] node name [es-hadoop13], node ID [6ga7KPpTQmiQtTlxo4plQQ]
[2017-07-13T23:40:00,739][INFO ][o.e.n.Node               ] [es-hadoop13] version[5.4.0], pid[2653], build[780f8c4/2017-04-28T17:43:27.229Z], OS[Linux/2.6.32-573.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_77/25.77-b03]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [aggs-matrix-stats]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [ingest-common]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [lang-expression]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [lang-groovy]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [lang-mustache]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [lang-painless]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [percolator]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [reindex]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [transport-netty3]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService     ] [es-hadoop13] loaded module [transport-netty4]
[2017-07-13T23:40:14,647][INFO ][o.e.p.PluginsService     ] [es-hadoop13] no plugins loaded
[2017-07-13T23:40:33,836][INFO ][o.e.d.DiscoveryModule    ] [es-hadoop13] using discovery type [zen]
[2017-07-13T23:40:36,207][INFO ][o.e.n.Node               ] [es-hadoop13] initialized
[2017-07-13T23:40:36,207][INFO ][o.e.n.Node               ] [es-hadoop13] starting ...
[2017-07-13T23:40:37,699][INFO ][o.e.t.TransportService   ] [es-hadoop13] publish_address {192.168.200.13:9300}, bound_addresses {192.168.200.13:9300}
[2017-07-13T23:40:37,709][INFO ][o.e.b.BootstrapChecks    ] [es-hadoop13] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
memory locking requested for elasticsearch process but memory is not locked
max number of threads [1024] for user [es] is too low, increase to at least [2048]
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-07-13T23:40:37,763][INFO ][o.e.n.Node               ] [es-hadoop13] stopping ...
[2017-07-13T23:40:37,830][INFO ][o.e.n.Node               ] [es-hadoop13] stopped
[2017-07-13T23:40:37,831][INFO ][o.e.n.Node               ] [es-hadoop13] closing ...
[2017-07-13T23:40:37,878][INFO ][o.e.n.Node               ] [es-hadoop13] closed
[es@hadoop13 root]$ vi /home/es/elasticsearch-5.4.0/config/elasticsearch.yml
[es@hadoop13 root]$

问题3:修改虚拟机及Linux环境

[es@hadoop12 config]$ /home/es/elasticsearch-5.4.0/bin/elasticsearch 
[2017-07-14T17:04:12,114][INFO ][o.e.n.Node               ] [es-hadoop12] initializing ...
[2017-07-14T17:04:13,332][INFO ][o.e.e.NodeEnvironment    ] [es-hadoop12] using [1] data paths, mounts [[/ (/dev/sda3)]], net usable_space [9.2gb], net total_space [15.4gb], spins? [possibly], types [ext4]
[2017-07-14T17:04:13,332][INFO ][o.e.e.NodeEnvironment    ] [es-hadoop12] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-07-14T17:04:13,333][INFO ][o.e.n.Node               ] [es-hadoop12] node name [es-hadoop12], node ID [laYcbaV_QLKGjW7uKaP5BQ]
[2017-07-14T17:04:13,334][INFO ][o.e.n.Node               ] [es-hadoop12] version[5.4.0], pid[7774], build[780f8c4/2017-04-28T17:43:27.229Z], OS[Linux/2.6.32-573.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_77/25.77-b03]
[2017-07-14T17:04:25,712][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [aggs-matrix-stats]
[2017-07-14T17:04:25,712][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [ingest-common]
[2017-07-14T17:04:25,712][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [lang-expression]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [lang-groovy]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [lang-mustache]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [lang-painless]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [percolator]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [reindex]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [transport-netty3]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService     ] [es-hadoop12] loaded module [transport-netty4]
[2017-07-14T17:04:25,715][INFO ][o.e.p.PluginsService     ] [es-hadoop12] no plugins loaded
[2017-07-14T17:04:39,353][INFO ][o.e.d.DiscoveryModule    ] [es-hadoop12] using discovery type [zen]
[2017-07-14T17:04:42,769][INFO ][o.e.n.Node               ] [es-hadoop12] initialized
[2017-07-14T17:04:42,770][INFO ][o.e.n.Node               ] [es-hadoop12] starting ...
[2017-07-14T17:04:44,195][INFO ][o.e.t.TransportService   ] [es-hadoop12] publish_address {192.168.200.12:9300}, bound_addresses {192.168.200.12:9300}
[2017-07-14T17:04:44,291][INFO ][o.e.b.BootstrapChecks    ] [es-hadoop12] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
max number of threads [1024] for user [es] is too low, increase to at least [2048]
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2017-07-14T17:04:44,358][INFO ][o.e.n.Node               ] [es-hadoop12] stopping ...
[2017-07-14T17:04:44,606][INFO ][o.e.n.Node               ] [es-hadoop12] stopped
[2017-07-14T17:04:44,606][INFO ][o.e.n.Node               ] [es-hadoop12] closing ...
[2017-07-14T17:04:44,682][INFO ][o.e.n.Node               ] [es-hadoop12] closed

问题4:max file descriptors

错误 max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
在linux系统:/etc/security/limits.conf
编辑该文件,后面加上:

xq soft nofile 65536
xq hard nofile 65536

备注:xq为用户名,同时确保xq用户拥有次命令权限

问题5:vm.max_map_count

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决:切换到root用户修改配置sysctl.conf

vi /etc/sysctl.conf 

添加下面配置:

vm.max_map_count=655360

并执行命令:

sysctl -p

然后,重新启动elasticsearch,即可启动成功。
或者:
是以为操作系统的vm.max_map_count参数设置太小导致的,请使用root用户登录系统,执行以下命令:

sysctl -w vm.max_map_count=655360

并用以下命令查看是否修改成功

sysctl -a | grep "vm.max_map_count"

如果能正常输出655360,则说明修改成功,然后再次启动elasticsearch
把配置好的安装包拷贝一份到其他两台机器上,修改 config/elasticsearch.yml下的node.name和network.host为对于的机器即可。

问题6:内存不足

max number of threads [1024] for user [es] likely too low, increase to at least [2048]
解决:切换到root用户,进入limits.d目录下修改配置文件。

vi /etc/security/limits.d/90-nproc.conf 

修改如下内容:

* soft nproc 1024

修改为

* soft nproc 2048

这里写图片描述
这里写图片描述

修改如下:

-Xms1g
-Xmx1g

问题7:Eclipse源码编译打包步骤

Run as ==> Maven clean
Run as ==> Maven build

1
0

查看评论
* 以上用户言论只代表其个人观点,不代表CSDN网站的观点或立场
    【个人简介】
    个人资料
    • 访问:362189次
    • 积分:6513
    • 等级:
    • 排名:第3939名
    • 原创:283篇
    • 转载:23篇
    • 译文:24篇
    • 评论:76条
    博客专栏