Ambari2.7.3+hdp3.1 集成Elasticsearch6.4.2

1.环境准备
1.1.环境说明
Ambari的集群环境是提前安装好的。
本次安装es使用的集群共有四个节点:
 node1.bde.unicloud.com
 node2.bde.unicloud.com
 node3.bde.unicloud.com
 node4.bde.unicloud.com

2.所需安装包

2.1.elasticsearch-6.4.2.tar.gz
下载地址：

https://www.elastic.co/cn/downloads/past-releases/elasticsearch-6-4-2

2.2.Ambari-Elastic-Service-master.zip
下载地址：

https://github.com/BalaBalaYi/Ambari-Elastic-Service
3.上传解压拷贝
3.1.Ambari-Elastic-Service-master.zip
3.1.1.新建/opt/es 目录

mkdir /opt/es
3.2.将Ambari-Elastic-Service-master.zip上传到/opt/es下解压
Unzip Ambari-Elastic-Service-master.zip
3.3.进入到解压后的目录，拷贝文件

Cd Ambari-Elastic-Service-master/

拷贝文件

cp -r ELASTICSEARCH-6.4.x /var/lib/ambari-server/resources/stacks/HDP/3.1/services/

进去到/var/lib/ambari-server/resources/stacks/HDP/3.1/services/目录

cd /var/lib/ambari-server/resources/stacks/HDP/3.1/services/

修改文件文件名称

mv ./ELASTICSEARCH-6.4.x ./ELASTICSEARCH

将下载的elasticsearch-6.4.2.tar.gz 移动到/var/www/html/es

新建/var/www/html/es 目录

mkdir /var/www/html/es

移动文件到改目录下

mv elasticsearch-6.4.2.tar.gz /var/www/html/es

通过浏览器输入：http://node1/es 测试能否访问此目录

3.4.重启 ambari-server
ambari-server restart
4.安装ES
添加服务

选择Elasticsearch

选择master 服务安装的节点

选择slave 服务安装的节点

服务配置界面提示需要补充完整

 安装完成

5.安装异常问题
5.1.用户名组问题

错误提示
KeyError: u’elasticsearch’
Error: Error: Unable to run the custom hook script [’/usr/bin/python’, ‘/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py’, ‘ANY’, ‘/var/lib/ambari-agent/data/command-102.json’, ‘/var/lib/ambari-agent/cache/stack-hooks/before-ANY’, ‘/var/lib/ambari-agent/data/structured-out-102.json’, ‘INFO’, ‘/var/lib/ambari-agent/tmp’, ‘PROTOCOL_TLSv1_2’, ‘’]
解决办法
需要修改ambari-server 资源中的配置，将gnore_groupsusers_create 的值由false改为true
cd /var/lib/ambari-server/resources/scripts

python configs.py -u admin -p admin -n c1 -l node1.bde.unicloud.com -t 8080 -a get -c cluster-env | grep -i ignore_groupsusers_create

python configs.py -u admin -p admin -n c1 -l node1.bde.unicloud.com -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true

手动创建es用户,所有master和slave节点都需要执行

useradd -g elasticsearch elasticsearch

5.2.无法导入format_hdp_stack_version

错误提示

ImportError: cannot import name format_hdp_stack_version
解决办法
进入/var/lib/ambari-server/resources/stacks/HDP/3.1/services/ELASTICSEARCH/package/scripts/目录
编辑params.py，将params.py文件中的format_hdp_stack_version删除

清除所有的elasticsearch slave 节点的ambari-agent 缓存

rm -rf /var/lib/ambari-agent/cache/*

Sever 节点重启

ambari-server restart

Slave 节点重启

ambari-agent restart

5.3. 安装出现警告
重启服务失败，报错

错误提示

File “/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ELASTICSEARCH/package/templates/elasticsearch.master.yml.j2”, line 93, in top-level template code
action.destructive_requires_name: {{action_destructive_requires_name}}
File “/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py”, line 73, in getattr
raise Fail(“Configuration parameter '” + self.name + “’ was not found in configurations dictionary!”)
resource_management.core.exceptions.Fail: Configuration parameter ‘hostname’ was not found in configurations dictionary!

解决办法

进入/var/lib/ambari-server/resources/stacks/HDP/3.1/services/ELASTICSEARCH/configuration/目录下

cd /var/lib/ambari-server/resources/stacks/HDP/3.1/services/ELASTICSEARCH/configuration/

编辑elasticsearch-config.xml，发现其中的discovery.zen.ping.unicast.hosts属性的值是空的。

修改discovery.zen.ping.unicast.hosts的值

discovery.zen.ping.unicast.hosts
node1

修改params.py

cd /var/lib/ambari-server/resources/stacks/HDP/3.1/services/ELASTICSEARCH/package/scripts/

vi params.py

修改hostname的值修改为：

hostname = config[‘configurations’][‘elasticsearch-config’][‘discovery.zen.ping.unicast.hosts’]

hostname有两处，两处都需要进行修改。

清除所有的elasticsearch slave 节点的ambari-agent 缓存

rm -rf /var/lib/ambari-agent/cache/*

Sever 节点重启

ambari-server restart

Slave 节点重启

ambari-agent restart

5.4.Master和salve 启动又停止
查看日志发现报错：

错误提示

max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

解决办法：

修改/etc/security/limits.conf文件，增加配置（所有master和slave节点）

错误提示

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

解决办法：

修改/etc/sysctl.conf文件，增加配置（所有master和slave节点）。

vm.max_map_count=262144
执行命令sysctl -p生效

sysctl -p

重启elasticsearch服务。

5.5.parent directory /opt/elasticsearch/master/config doesnot exist

这个错误是在slave节点上出现的

解决办法：创建一个空的目录

mkdir -p /opt/elasticsearch/master/config

重启elasticsearch服务。

6.安装完成
最终如下图所示

测试es：

浏览器中打开http://node1:9200 ，出现如下界面