Ambari2.75集成flume1.9

Ambari2.75默认是没有flume组件，这里需要自定义集成。（ambari的安装可参考我上一篇文章：最新版Ambari2.75安装及HDP3.1.5集群搭建）

一：使用如下github地址，将编译的包，以及将flume添加到ambari-server的web页中的包，克隆到ambari-server所在服务器的一个目录中。

git clone https://github.com/maikoulin/ambari-flume-service.git

二：去官网下载flume的tar包：https://flume.apache.org/download.html。下个 apache-flume-1.9.0-bin.tar.gz 的tar.gz包，或者直接使用如下命令：

wget https://dlcdn.apache.org/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz

三：将apache-flume-1.9.0-bin.tar.gz 放到ambari-flume-service/buildrpm/rpmbuild/SOURCES 这个相对目录下。

四: ambari-flume-service/buildrpm 目录下执行:

sh buildrpm.sh

到此，rpm包编译完成。

五：将 flumerpm/ambari-flume-service 相对目录下的FLUME 复制到 ambari-server 主机的 /var/lib/ambari-server/resources/stacks/HDP/3.1/services/ 目录下并重启ambari-server:

ambari-server restart

到此，ambari的web界面就能识别出来flume了。

六：创建flume的本地yum源

关于本地yum源的设置可参考我上一篇文章：最新版Ambari2.75安装及HDP3.1.5集群搭建

mkdir /var/www/html/flume/

// 创建yum源

createrepo /var/www/html/flume/

// 将上面生成的rpm包拷贝到此

cp ambari-flume-service/buildrpm/rpmbuild/RPMS/noarch/flume-1.9.0-1.el7.noarch.rpm /var/www/html/flume/

七、创建repo

cd /etc/yum.repos.d/

cp centos.repo flume.repo

vim flume.repo 

[flume-1.9.0]
name=flume-1.9.0
baseurl=http://hadoop01/flume/
gpgcheck=0
enabled=1

八、将flume.repo复制到需要安装的子节点上

scp flume.repo hadoop05:/etc/yum.repos.d/

九：通过ambariUI安装flume

1）点击 add service，勾选flume

点击 next。

2） 分配客户端

点击next。

3） 填写flume agent配置文件

可以在这里填写配置，也可以不填，等安装完成再填写。点击next。 

4）预览页面

直接点击 deploy。

 5）安装成功

点击next。 

6）完成

点击complete。

 7）填写agent配置文件

如果刚刚在安装过程过程中没有填写 配置文件，可在此点击 “Configure Agents”，填写配置文件：

（1）通过avro方式接受数据，最后通过日志形式输出。

logger.sources = r1
logger.sinks = k1
logger.channels = c1

# Describe/configure the source
logger.sources.r1.type = avro
logger.sources.r1.bind = 0.0.0.0
logger.sources.r1.port = 6666

# Describe the sink
logger.sinks.k1.channel=c1
logger.sinks.k1.type=logger

#Spillable Memory Channel
logger.channels.c1.type=SPILLABLEMEMORY
logger.channels.c1.checkpointDir = /data/flume/checkpoint
logger.channels.c1.dataDirs = /data/flume

# Bind the source and sink to the channel
logger.sources.r1.channels = c1

（2）通过avro方式接受数据，输出到kafka。

logger.sources = r1
logger.sinks = k1
logger.channels = c1

# Describe/configure the source
logger.sources.r1.type = avro
logger.sources.r1.bind = 0.0.0.0
logger.sources.r1.port = 6666

# Describe the sink
logger.sinks.k1.channel=c1
logger.sinks.k1.type=org.apache.flume.sink.kafka.KafkaSink
logger.sinks.k1.brokerList=hadoop03:6667,hadoop04:6667,hadoop05:6667
logger.sinks.k1.topic=buriedLogger
logger.sinks.k1.serializer.class=kafka.serializer.StringEncoder
logger.sinks.k1.serializer.appendNewline=false

#Spillable Memory Channel
logger.channels.c1.type=SPILLABLEMEMORY
logger.channels.c1.checkpointDir = /data/flume/checkpoint
logger.channels.c1.dataDirs = /data/flume

# Bind the source and sink to the channel
logger.sources.r1.channels = c1

保存之后，点击“ACTIONS”>“Restart All”。到此安装完成。

问题：

1.Failed to execute command '/usr/bin/yum -y install flume', exited with code '1', message: 'Error: Nothing to do'

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLUME/package/scripts/flume_handler.py", line 134, in <module>
    FlumeHandler().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLUME/package/scripts/flume_handler.py", line 53, in install
    self.install_packages(env)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 853, in install_packages
    retry_count=agent_stack_retry_count)
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/packaging.py", line 30, in action_install
    self._pkg_manager.install_package(package_name, self.__create_context())
  File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 219, in install_package
    shell.repository_manager_executor(cmd, self.properties, context)
  File "/usr/lib/ambari-agent/lib/ambari_commons/shell.py", line 753, in repository_manager_executor
    raise RuntimeError(message)
RuntimeError: Failed to execute command '/usr/bin/yum -y install flume', exited with code '1', message: 'Error: Nothing to do'

解决办法：没有配置好flume的yum源，按照六、七、八步骤操作即可。一定要把flume.repo复制到需要安装的子节点上，比如：要在hadoop01、hadoop02节点安装flume，则需要把flume.repo复制到以上两个节点。

2.Package flume-1.9.0-1.el7.noarch.rpm is not signed

解决办法：由于flume的rpm包时在本地编译的，没有对应的签名，将flume.repo中的gpgcheck设置为0即可。 gpgcheck表示使用gpg文件来检查软件包的签名。

3.安装了flume的节点spark-shell启动报错：

hdfs@hadoop05 spark2-client]$ ./bin/spark-shell 
Traceback (most recent call last):
  File "/bin/hdp-select", line 453, in <module>
    listPackages(getPackages("all"))
  File "/bin/hdp-select", line 271, in listPackages
    os.path.basename(os.path.dirname(os.readlink(linkname))))
OSError: [Errno 22] Invalid argument: '/usr/hdp/current/flume-server'
ls: cannot access /usr/hdp//hadoop/lib: No such file or directory
Exception in thread "main" java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx
	at org.apache.spark.launcher.Main.main(Main.java:118)

 解决办法：将/usr/hdp/current下的flume-server文件夹复制到/usr/hdp/3.1.5.0-152下，然后将/usr/hdp/current下的flume-server文件夹删除，创建软链接过去
 

cd /usr/hdp/current

cp -r flume-server ../3.1.5.0-152/

rm -rf flume-server

ln -s /usr/hdp/3.1.5.0-152/flume-server flume-server

4、后续使用发现通过ambari启动flume，flume的启动日志（/var/log/flume/${agent_name}.out）不显示

解决办法：

在ambari-server所在机器上执行如下命令：

vim /var/lib/ambari-server/resources/stacks/HDP/3.1/services/FLUME/package/scripts/flume.py

找到 conf-file 所在位置，添加 -Dflume.root.logger=INFO,console 如下图：

然后重启ambari-server，重启flume即可。