Ambari2.7.5自定义服务集成Flink1.7.2

ambari支持自定义服务组件集成，以下介绍ambari2.7.5集成flink1.7.2版本组件。

flink：  

  https://github.com/abajwa-hw/ambari-flink-service

  https://archive.apache.org/dist/flink/flink-1.7.2/flink-1.7.2-bin-scala_2.11.tgz

首先设置一下HADOOP版本：

VERSION=`hdp-select status hadoop-client | sed's/hadoop-client - \([0-9]\.[0-9]\).*/\1/'`

查看一下版本是否设置成功

echo ${VERSION}

如下图：我这边是 3.1版本的

下载ambari-flink-service服务

（如果下载有问题 可以使用其他方式下载 然后传节点上）

sudo git clone https://github.com/abajwa-hw/ambari-flink-service.git   /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/FLINK   

我这边下载的嘎嘎快 如下图

如果开启了kerberos 需要配置kerberos.json 其原理可以查看下 KerberosOperationHandler 这个接口类 随后单独出个文章讲下。

本文没有开启Kerberos 。随后更新一版。

修改metainfo.xml：

修改flink版本

            <name>FLINK</name>
            <displayName>Flink</displayName>
            <comment>Apache Flink is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.</comment>
            <version>1.7.2</version>

修改flink运行模式

<name>FLINK_CLIENT</name>
<displayName>Flink</displayName>
<category>CLIENT</category>

添加flink on yarn 配置

<property>
    <name>yarn.client.failover-proxy-provider</name>
    <value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>
</property>

注意这个位置

配置Flink下载地址

vim configuration/flink-ambari-config.xml 我这里写了本地http上的

<property>
        <name>flink_download_url</name>
        <value>http://hadoop01/flink-1.7.2-bin-scala_2.11.gz</value>
        <description>Snapshot download location. Downloaded when setup_prebuilt is true</description>
</property>

添加用户组(所有节点执行)

groupadd flink
useradd -d /home/flink -g flink flink

重启ambari-server

ambari-server restart

然后界面添加flink服务

安装完成：

后台启动yarn-session看看：

/usr/hdp/3.1.5.0-152/flink/bin/yarn-session.sh -d -nm flinkapp-from-ambari -n 1 -s 1 -jm 768 -tm 1024 -qu default

启动后查看yarn界面：

运行wordcount测试：

flink run WordCount.jar

完成 ok

FAQ～～～

install的时候发生的问题

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 38, in <module>
    BeforeAnyHook().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 31, in hook
    setup_users()
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py", line 50, in setup_users
    groups = params.user_to_groups_dict[user],
KeyError: u'flink'
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-153.json', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-153.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1_2', '']

解决方法：

cd /var/lib/ambari-server/resources/scripts

查看下ignore_groupsusers_create配置：

python configs.py -u admin -p admin -n test -l hadoop01 -t 8080 -a get -c cluster-env |grep -i ignore_groupsusers_create

更改为true

python configs.py -u admin -p admin -n test -l hadoop01 -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true

报错找不到这个配置文件

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLINK/package/scripts/flink.py", line 172, in <module>
    Master().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLINK/package/scripts/flink.py", line 108, in start
    self.configure(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLINK/package/scripts/flink.py", line 91, in configure
    File(format("{conf_dir}/flink-conf.yaml"), content=properties_content, owner=params.flink_user)
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 120, in action_create
    raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
resource_management.core.exceptions.Fail: Applying File['/opt/flink/conf/flink-conf.yaml'] failed, parent directory /opt/flink/conf doesn't exist

他去找配置文件发现不存在

我这边创建了这个目录 把配置放进去了