ambari支持自定义服务组件集成,以下介绍ambari2.7.5集成flink1.7.2版本组件。
flink:
首先设置一下HADOOP版本:
VERSION=`hdp-select status hadoop-client | sed's/hadoop-client - \([0-9]\.[0-9]\).*/\1/'`
查看一下版本是否设置成功
echo ${VERSION}
如下图:我这边是 3.1版本的
![](https://i-blog.csdnimg.cn/blog_migrate/0356bab98e0b71b540d90fa6fc13fae2.png)
下载ambari-flink-service服务
(如果下载有问题 可以使用其他方式下载 然后传节点上)
sudo git clone https://github.com/abajwa-hw/ambari-flink-service.git /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/FLINK
我这边下载的嘎嘎快 如下图
![](https://i-blog.csdnimg.cn/blog_migrate/6ba0e5467a10eab0404c725ebdfe441e.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5c0864850626ba75822970c326c1ed80.png)
如果开启了kerberos 需要配置kerberos.json 其原理可以查看下
KerberosOperationHandler 这个接口类 随后单独出个文章讲下。
本文没有开启Kerberos 。随后更新一版。
修改metainfo.xml:
修改flink版本
<name>FLINK</name>
<displayName>Flink</displayName>
<comment>Apache Flink is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.</comment>
<version>1.7.2</version>
修改flink运行模式
<name>FLINK_CLIENT</name>
<displayName>Flink</displayName>
<category>CLIENT</category>
添加flink on yarn 配置
<property>
<name>yarn.client.failover-proxy-provider</name>
<value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>
</property>
注意这个位置
![](https://i-blog.csdnimg.cn/blog_migrate/8a36d66c596060b4961d4d266f4d10b8.png)
配置Flink下载地址
vim configuration/flink-ambari-config.xml 我这里写了本地http上的
<property>
<name>flink_download_url</name>
<value>http://hadoop01/flink-1.7.2-bin-scala_2.11.gz</value>
<description>Snapshot download location. Downloaded when setup_prebuilt is true</description>
</property>
添加用户组(所有节点执行)
groupadd flink
useradd -d /home/flink -g flink flink
重启ambari-server
ambari-server restart
![](https://i-blog.csdnimg.cn/blog_migrate/e316226dbd2f9a032f7b61dea411a08f.png)
然后界面添加flink服务
![](https://i-blog.csdnimg.cn/blog_migrate/03e276f0ef9555b1760bf3bf1b172afc.png)
安装完成:
![](https://i-blog.csdnimg.cn/blog_migrate/12f187f96917aab7bb79f62db9bc40ba.png)
后台启动yarn-session看看:
/usr/hdp/3.1.5.0-152/flink/bin/yarn-session.sh -d -nm flinkapp-from-ambari -n 1 -s 1 -jm 768 -tm 1024 -qu default
启动后查看yarn界面:
![](https://i-blog.csdnimg.cn/blog_migrate/9ef996467dc5aac827ba00b1bc13aa0b.png)
运行wordcount测试:
flink run WordCount.jar
![](https://i-blog.csdnimg.cn/blog_migrate/78dbb44def4eb68ecb7d077023001498.png)
完成 ok
![](https://i-blog.csdnimg.cn/blog_migrate/35cafbdb86b25fcb6370f406809b781b.png)
FAQ~~~
install的时候发生的问题
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 38, in <module>
BeforeAnyHook().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 31, in hook
setup_users()
File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py", line 50, in setup_users
groups = params.user_to_groups_dict[user],
KeyError: u'flink'
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-153.json', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-153.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1_2', '']
解决方法:
cd /var/lib/ambari-server/resources/scripts
查看下ignore_groupsusers_create配置:
python configs.py -u admin -p admin -n test -l hadoop01 -t 8080 -a get -c cluster-env |grep -i ignore_groupsusers_create
![](https://i-blog.csdnimg.cn/blog_migrate/c454987716015c35254c66bb10c79cde.png)
更改为true
python configs.py -u admin -p admin -n test -l hadoop01 -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true
![](https://i-blog.csdnimg.cn/blog_migrate/d6003216a407b31f5231ce778db99c31.png)
报错找不到这个配置文件
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLINK/package/scripts/flink.py", line 172, in <module>
Master().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLINK/package/scripts/flink.py", line 108, in start
self.configure(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLINK/package/scripts/flink.py", line 91, in configure
File(format("{conf_dir}/flink-conf.yaml"), content=properties_content, owner=params.flink_user)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 120, in action_create
raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
resource_management.core.exceptions.Fail: Applying File['/opt/flink/conf/flink-conf.yaml'] failed, parent directory /opt/flink/conf doesn't exist
他去找配置文件发现不存在
我这边创建了这个目录 把配置放进去了
![](https://i-blog.csdnimg.cn/blog_migrate/046eddc4974b245ec64d2b5032dd7049.png)