1、前期准备
2、集群搭建
1)从官网下载ES 安装包
2)新增ES用户
3)上传解压ES
4)配置ES
5)启动前提前期解决的问题
6)启动、关闭ES
7)集群验证
8)安装插件(kibana-5.4.0-linux-x86_64)
9)集群验证
3、集群搭建出现的问题
问题1:Maven有问题包下载不下来
问题2:配置错误
问题3:修改虚拟机及Linux环境
问题4:max file descriptors
问题5:vm.max_map_count
问题6:内存不足
问题7:Eclipse源码编译打包步骤
————————————————————————————-
1、前期准备
下载地址:
Java程序员软件包
http://blog.csdn.net/silentwolfyh/article/details/50936377
大数据程序员软件包
http://blog.csdn.net/silentwolfyh/article/details/50936459
Java版本:java version “1.8.0_77”
系统版本:Centos 6.7
机器节点:hadoop11,hadoop12,hadoop13
软件:
elasticsearch-5.4.0.tar.gz
elasticsearch-analysis-ik-5.4.0.zip
kibana-5.4.0-linux-x86_64.tar.gz
2、集群搭建
1)从官网下载ES 安装包
elasticsearch-5.4.0.tar.gz
2)新增ES用户
注意: Linux新增,用户用户名为es,密码为hadoop
[root@hadoop11 ~]# adduser es
[root@hadoop11 ~]# passwd es
更改用户 es 的密码 。
新的 密码:
无效的密码: 它基于字典单词
无效的密码: 过于简单
重新输入新的 密码:
passwd: 所有的身份验证令牌已经成功更新。
[root@hadoop11 ~]# su es
[es@hadoop11 root]$
3)上传解压ES
注意:一定要切换用户,不能用root用户解压,不能用root用户启动,ES官网规定的。
[es@hadoop11 root]cd/usr/app/[es@hadoop11 ] pwd
/home/es
[es@hadoop11 es]$ tar -zxvf elasticsearch-5.4.0.tar.gz
4)配置ES
注意:第3个节点都要修改node.name
编辑文档如下:
[es@hadoop11 root]$ vi /home/es/elasticsearch-5.4.0/config/elasticsearch.yml
文档配置解说
cluster.name: es-cluster #集群名称
node.name: es-hadoop11 #节点名称
path.data: /home/es/elasticsearch-5.4.0/data #数据文件存储路径
path.logs: /home/es/elasticsearch-5.4.0/logs #log文件存储路径
network.host: 192.168.200.11 #默认网络连接地址
http.port: 9200 #连接端口
discovery.zen.minimum_master_nodes: 2 #这个参数来保证集群中的节点可以知道其它N个有master资格的节点。默认为1,对于大的集群来说,可以设置大一点的值(2-4)
discovery.zen.ping.unicast.hosts: [“192.168.7.126”, “192.168.7.125”] #集群中master节点的初始列表,可以通过这些节点来自动发现新加入集群的节点。
discovery.zen.ping_timeout: 120s
bootstrap.system_call_filter: false
http.cors.enabled: true
http.cors.allow-origin: “*”
文档主要不同的配置
Hadoop11
cluster.name: es-cluster
node.name: es-hadoop11
path.data: /home/es/elasticsearch-5.4.0/data
path.logs: /home/es/elasticsearch-5.4.0/logs
network.host: 192.168.200.11
http.port: 9200
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["192.168.200.11", "192.168.200.12", "192.168.200.13"]
discovery.zen.ping_timeout: 120s
bootstrap.system_call_filter: false
http.cors.enabled: true
http.cors.allow-origin: "*"
Hadoop12
cluster.name: es-cluster
node.name: es-hadoop12
path.data: /home/es/elasticsearch-5.4.0/data
path.logs: /home/es/elasticsearch-5.4.0/logs
network.host: 192.168.200.12
http.port: 9200
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["192.168.200.11", "192.168.200.12", "192.168.200.13"]
discovery.zen.ping_timeout: 120s
bootstrap.system_call_filter: false
http.cors.enabled: true
http.cors.allow-origin: "*"
Hadoop13
cluster.name: es-cluster
node.name: es-hadoop13
path.data: /home/es/elasticsearch-5.4.0/data
path.logs: /home/es/elasticsearch-5.4.0/logs
network.host: 192.168.200.13
http.port: 9200
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["192.168.200.11", "192.168.200.12", "192.168.200.13"]
discovery.zen.ping_timeout: 120s
bootstrap.system_call_filter: false
http.cors.enabled: true
http.cors.allow-origin: "*"
5)启动前提前期解决的问题
问题:max file descriptors
问题:vm.max_map_count
问题:内存不足
6)启动、关闭ES
/home/es/elasticsearch-5.4.0/bin/elasticsearch ---前台启动
/home/es/elasticsearch-5.4.0/bin/elasticsearch -d ---后台启动
查看启动进程:jps | grep Elasticsearch
关闭:
kill -15 pid
7)集群验证
http://hadoop11:9200/
8)安装插件(kibana-5.4.0-linux-x86_64)
【注意】用es用户解压,修改kibana-5.4.0-linux-x86_64/config/kibana.yml
修改配置
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.200.11:9200"
执行 bin/kibana 即可
注意:
1. 需要将server.host修改一下,否则远程无法访问,这个配置在Kibana4.6.1(对应ES2.4.0)的时候还不需要配置,但是5.4.0就需要了
2. 之前ES和Kibana配合还需要使用plugin安装一些Marvel,sense等,现在都不需要了,DevTools就是之前的Sense
9)集群验证
GET /_cat/health?v
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1500041888 22:18:08 es-cluster green 3 3 2 1 0 0 0 0 - 100.0%
如何快速了解集群的健康状况?green、yellow、red?
green:每个索引的primary shard和replica shard都是active状态的
yellow:每个索引的primary shard都是active状态的,但是部分replica shard不是active状态,处于不可用的状态
red:不是所有索引的primary shard都是active状态的,部分索引有数据丢失了
为什么现在会处于一个yellow状态?
我们现在就一个笔记本电脑,就启动了一个es进程,相当于就只有一个node。现在es中有一个index,就是kibana自己内置建立的index。由于默认的配置是给每个index分配5个primary shard和5个replica shard,而且primary shard和replica shard不能在同一台机器上(为了容错)。现在kibana自己建立的index是1个primary shard和1个replica shard。当前就一个node,所以只有1个primary shard被分配了和启动了,但是一个replica shard没有第二台机器去启动。
做一个小实验:此时只要启动第二个es进程,就会在es集群中有2个node,然后那1个replica shard就会自动分配过去,然后cluster status就会变成green状态。
3、集群搭建出现的问题
问题1:Maven有问题包下载不下来
点击项目 —> 右键 —> maven —> Update Project
问题2:配置错误
https://github.com/elastic/elasticsearch/issues/22899
bootstrap.system_call_filter: true
[es@hadoop13 root]$ /home/es/elasticsearch-5.4.0/bin/elasticsearch
[2017-07-13T23:39:55,695][WARN ][o.e.b.JNANatives ] unable to install syscall filter:
java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:363) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:638) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:215) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:99) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.0.jar:5.4.0]
[2017-07-13T23:39:55,805][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=无法分配内存
[2017-07-13T23:39:55,806][WARN ][o.e.b.JNANatives ] This can result in part of the JVM being swapped out.
[2017-07-13T23:39:55,807][WARN ][o.e.b.JNANatives ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2017-07-13T23:39:55,807][WARN ][o.e.b.JNANatives ] These can be adjusted by modifying /etc/security/limits.conf, for example:
# allow user 'es' mlockall
es soft memlock unlimited
es hard memlock unlimited
[2017-07-13T23:39:55,807][WARN ][o.e.b.JNANatives ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2017-07-13T23:39:59,020][INFO ][o.e.n.Node ] [es-hadoop13] initializing ...
[2017-07-13T23:40:00,737][INFO ][o.e.e.NodeEnvironment ] [es-hadoop13] using [1] data paths, mounts [[/ (/dev/sda3)]], net usable_space [9.3gb], net total_space [15.4gb], spins? [possibly], types [ext4]
[2017-07-13T23:40:00,737][INFO ][o.e.e.NodeEnvironment ] [es-hadoop13] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-07-13T23:40:00,739][INFO ][o.e.n.Node ] [es-hadoop13] node name [es-hadoop13], node ID [6ga7KPpTQmiQtTlxo4plQQ]
[2017-07-13T23:40:00,739][INFO ][o.e.n.Node ] [es-hadoop13] version[5.4.0], pid[2653], build[780f8c4/2017-04-28T17:43:27.229Z], OS[Linux/2.6.32-573.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_77/25.77-b03]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService ] [es-hadoop13] loaded module [aggs-matrix-stats]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService ] [es-hadoop13] loaded module [ingest-common]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService ] [es-hadoop13] loaded module [lang-expression]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService ] [es-hadoop13] loaded module [lang-groovy]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService ] [es-hadoop13] loaded module [lang-mustache]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService ] [es-hadoop13] loaded module [lang-painless]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService ] [es-hadoop13] loaded module [percolator]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService ] [es-hadoop13] loaded module [reindex]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService ] [es-hadoop13] loaded module [transport-netty3]
[2017-07-13T23:40:14,646][INFO ][o.e.p.PluginsService ] [es-hadoop13] loaded module [transport-netty4]
[2017-07-13T23:40:14,647][INFO ][o.e.p.PluginsService ] [es-hadoop13] no plugins loaded
[2017-07-13T23:40:33,836][INFO ][o.e.d.DiscoveryModule ] [es-hadoop13] using discovery type [zen]
[2017-07-13T23:40:36,207][INFO ][o.e.n.Node ] [es-hadoop13] initialized
[2017-07-13T23:40:36,207][INFO ][o.e.n.Node ] [es-hadoop13] starting ...
[2017-07-13T23:40:37,699][INFO ][o.e.t.TransportService ] [es-hadoop13] publish_address {192.168.200.13:9300}, bound_addresses {192.168.200.13:9300}
[2017-07-13T23:40:37,709][INFO ][o.e.b.BootstrapChecks ] [es-hadoop13] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
memory locking requested for elasticsearch process but memory is not locked
max number of threads [1024] for user [es] is too low, increase to at least [2048]
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-07-13T23:40:37,763][INFO ][o.e.n.Node ] [es-hadoop13] stopping ...
[2017-07-13T23:40:37,830][INFO ][o.e.n.Node ] [es-hadoop13] stopped
[2017-07-13T23:40:37,831][INFO ][o.e.n.Node ] [es-hadoop13] closing ...
[2017-07-13T23:40:37,878][INFO ][o.e.n.Node ] [es-hadoop13] closed
[es@hadoop13 root]$ vi /home/es/elasticsearch-5.4.0/config/elasticsearch.yml
[es@hadoop13 root]$
问题3:修改虚拟机及Linux环境
[es@hadoop12 config]$ /home/es/elasticsearch-5.4.0/bin/elasticsearch
[2017-07-14T17:04:12,114][INFO ][o.e.n.Node ] [es-hadoop12] initializing ...
[2017-07-14T17:04:13,332][INFO ][o.e.e.NodeEnvironment ] [es-hadoop12] using [1] data paths, mounts [[/ (/dev/sda3)]], net usable_space [9.2gb], net total_space [15.4gb], spins? [possibly], types [ext4]
[2017-07-14T17:04:13,332][INFO ][o.e.e.NodeEnvironment ] [es-hadoop12] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-07-14T17:04:13,333][INFO ][o.e.n.Node ] [es-hadoop12] node name [es-hadoop12], node ID [laYcbaV_QLKGjW7uKaP5BQ]
[2017-07-14T17:04:13,334][INFO ][o.e.n.Node ] [es-hadoop12] version[5.4.0], pid[7774], build[780f8c4/2017-04-28T17:43:27.229Z], OS[Linux/2.6.32-573.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_77/25.77-b03]
[2017-07-14T17:04:25,712][INFO ][o.e.p.PluginsService ] [es-hadoop12] loaded module [aggs-matrix-stats]
[2017-07-14T17:04:25,712][INFO ][o.e.p.PluginsService ] [es-hadoop12] loaded module [ingest-common]
[2017-07-14T17:04:25,712][INFO ][o.e.p.PluginsService ] [es-hadoop12] loaded module [lang-expression]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService ] [es-hadoop12] loaded module [lang-groovy]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService ] [es-hadoop12] loaded module [lang-mustache]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService ] [es-hadoop12] loaded module [lang-painless]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService ] [es-hadoop12] loaded module [percolator]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService ] [es-hadoop12] loaded module [reindex]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService ] [es-hadoop12] loaded module [transport-netty3]
[2017-07-14T17:04:25,713][INFO ][o.e.p.PluginsService ] [es-hadoop12] loaded module [transport-netty4]
[2017-07-14T17:04:25,715][INFO ][o.e.p.PluginsService ] [es-hadoop12] no plugins loaded
[2017-07-14T17:04:39,353][INFO ][o.e.d.DiscoveryModule ] [es-hadoop12] using discovery type [zen]
[2017-07-14T17:04:42,769][INFO ][o.e.n.Node ] [es-hadoop12] initialized
[2017-07-14T17:04:42,770][INFO ][o.e.n.Node ] [es-hadoop12] starting ...
[2017-07-14T17:04:44,195][INFO ][o.e.t.TransportService ] [es-hadoop12] publish_address {192.168.200.12:9300}, bound_addresses {192.168.200.12:9300}
[2017-07-14T17:04:44,291][INFO ][o.e.b.BootstrapChecks ] [es-hadoop12] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
max number of threads [1024] for user [es] is too low, increase to at least [2048]
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2017-07-14T17:04:44,358][INFO ][o.e.n.Node ] [es-hadoop12] stopping ...
[2017-07-14T17:04:44,606][INFO ][o.e.n.Node ] [es-hadoop12] stopped
[2017-07-14T17:04:44,606][INFO ][o.e.n.Node ] [es-hadoop12] closing ...
[2017-07-14T17:04:44,682][INFO ][o.e.n.Node ] [es-hadoop12] closed
问题4:max file descriptors
错误 max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
在linux系统:/etc/security/limits.conf
编辑该文件,后面加上:
xq soft nofile 65536
xq hard nofile 65536
备注:xq为用户名,同时确保xq用户拥有次命令权限
问题5:vm.max_map_count
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决:切换到root用户修改配置sysctl.conf
vi /etc/sysctl.conf
添加下面配置:
vm.max_map_count=655360
并执行命令:
sysctl -p
然后,重新启动elasticsearch,即可启动成功。
或者:
是以为操作系统的vm.max_map_count参数设置太小导致的,请使用root用户登录系统,执行以下命令:
sysctl -w vm.max_map_count=655360
并用以下命令查看是否修改成功
sysctl -a | grep "vm.max_map_count"
如果能正常输出655360,则说明修改成功,然后再次启动elasticsearch
把配置好的安装包拷贝一份到其他两台机器上,修改 config/elasticsearch.yml下的node.name和network.host为对于的机器即可。
问题6:内存不足
max number of threads [1024] for user [es] likely too low, increase to at least [2048]
解决:切换到root用户,进入limits.d目录下修改配置文件。
vi /etc/security/limits.d/90-nproc.conf
修改如下内容:
* soft nproc 1024
修改为
* soft nproc 2048
修改如下:
-Xms1g
-Xmx1g
问题7:Eclipse源码编译打包步骤
Run as ==> Maven clean
Run as ==> Maven build