Flume高可用集群搭建

9.5 Flume 安装

9.5.1 flume下载

# cd /opt
# wget https://mirrors.huaweicloud.com/apache/flume/1.8.0/apache-flume-1.8.0-bin.tar.gz
# tar -zxvf apache-flume-1.9.0-bin.tar.gz 
# mv apache-flume-1.9.0-bin/ flume

删除lib/guava-11.0.2.jar 防止不兼容

# cd /opt/flume/lib
# rm -rf guava-11.0.2.jar 

9.5.2 配置环境变量

# vi /etc/profile

export FLUME_HOME=/opt/flume/bin
export PATH=$PATH:$FLUME_HOME/bin

# source /etc/profile

拷贝到其他虚拟机

# scp /etc/profile root@hadoop02:/etc
# scp /etc/profile root@hadoop03:/etc
# scp /etc/profile root@hadoop04:/etc
# scp /etc/profile root@hadoop05:/etc

修改flume配置

# cd /opt/flume/conf

//加入
export JAVA_HOME=/opt/java/jdk1.8.0_141

9.5.3 监听端口测试

• 通过netcat工具向本机44444端口发送数据
• flume监控本机44444端口，通过flume的source端读取数据
• flume将获得的数据通过sink端写到控制台

安装netcat工具

# yum install -y nc

在flume中执行

# mkdir /opt/flume/job
# cd /opt/flume/job
# touch net-flume-logger.conf

粘贴如下配置

# example.conf: A single-node Flume configuration

# Name the components on this agent  这个名字在单台flume唯一
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

在本机执行

# bin/flume-ng agent -n a1 -c conf/ -f job/net-flume-logger.conf -Dflume.root.logger=INFO,console

复制一份本机ssh

# nc localhost 44444

就可以在本机通信了

拷贝到其他机器

# cd /opt
# scp -r flume hadoop02:$PWD
# scp -r flume hadoop03:$PWD
# scp -r flume hadoop04:$PWD
# scp -r flume hadoop05:$PWD