Ambari2.7.5自定义服务集成Flink1.7.2

ambari支持自定义服务组件集成,以下介绍ambari2.7.5集成flink1.7.2版本组件。
flink:  

首先设置一下HADOOP版本:

VERSION=`hdp-select status hadoop-client | sed's/hadoop-client - \([0-9]\.[0-9]\).*/\1/'`

查看一下版本是否设置成功
echo ${VERSION}

如下图:我这边是 3.1版本的

下载ambari-flink-service服务

(如果下载有问题 可以使用其他方式下载 然后传节点上)
sudo git clone https://github.com/abajwa-hw/ambari-flink-service.git   /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/FLINK   

我这边下载的嘎嘎快 如下图
如果开启了kerberos 需要配置kerberos.json 其原理可以查看下 KerberosOperationHandler 这个接口类 随后单独出个文章讲下。
本文没有开启Kerberos 。随后更新一版。

修改metainfo.xml:

修改flink版本

            <name>FLINK</name>
            <displayName>Flink</displayName>
            <comment>Apache Flink is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.</comment>
            <version>1.7.2</version>

修改flink运行模式

<name>FLINK_CLIENT</name>
<displayName>Flink</displayName>
<category>CLIENT</category>

添加flink on yarn 配置

<property>
    <name>yarn.client.failover-proxy-provider</name>
    <value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>
</property>
注意这个位置

配置Flink下载地址

vim configuration/flink-ambari-config.xml 我这里写了本地http上的
<property>
        <name>flink_download_url</name>
        <value>http://hadoop01/flink-1.7.2-bin-scala_2.11.gz</value>
        <description>Snapshot download location. Downloaded when setup_prebuilt is true</description>
</property>

添加用户组(所有节点执行)

groupadd flink
useradd -d /home/flink -g flink flink

重启ambari-server

ambari-server restart

然后界面添加flink服务

安装完成:

后台启动yarn-session看看:

/usr/hdp/3.1.5.0-152/flink/bin/yarn-session.sh -d -nm flinkapp-from-ambari -n 1 -s 1 -jm 768 -tm 1024 -qu default

启动后查看yarn界面:

运行wordcount测试:

flink run WordCount.jar

完成 ok

FAQ~~~

install的时候发生的问题
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 38, in <module>
    BeforeAnyHook().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 31, in hook
    setup_users()
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py", line 50, in setup_users
    groups = params.user_to_groups_dict[user],
KeyError: u'flink'
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-153.json', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-153.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1_2', '']

解决方法:
cd /var/lib/ambari-server/resources/scripts

查看下ignore_groupsusers_create配置:
python configs.py -u admin -p admin -n test -l hadoop01 -t 8080 -a get -c cluster-env |grep -i ignore_groupsusers_create

更改为true
python configs.py -u admin -p admin -n test -l hadoop01 -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true

报错找不到这个配置文件
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLINK/package/scripts/flink.py", line 172, in <module>
    Master().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLINK/package/scripts/flink.py", line 108, in start
    self.configure(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLINK/package/scripts/flink.py", line 91, in configure
    File(format("{conf_dir}/flink-conf.yaml"), content=properties_content, owner=params.flink_user)
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 120, in action_create
    raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
resource_management.core.exceptions.Fail: Applying File['/opt/flink/conf/flink-conf.yaml'] failed, parent directory /opt/flink/conf doesn't exist

他去找配置文件发现不存在
我这边创建了这个目录 把配置放进去了

  • 0
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值