大数据-Flume(三)：案例、配置文件编写【案例①：netcat--＞logger】【案例②：taildir--＞hdfs】

一、netcat-->logger

1、案例需求：

使用Flume监听一个端口，收集该端口数据，并打印到控制台。

2、需求分析

3、实现步骤

3.1 软件环境配置

（1）安装netcat工具

[wyr@hadoop102 ~]$ sudo yum install -y nc

（2）判断44444端口是否被占用

[wyr@hadoop102 ~]$ sudo netstat -nlp | grep 44444

（3）在flume目录下创建job文件夹并进入job文件夹。

[wyr@hadoop102 flume]$ mkdir -p job/simpleCase
[wyr@hadoop102 flume]$ cd job/simpleCase

3.2 编写配置文件

在job/simpleCase文件夹下创建Flume Agent配置文件flume-1-netcat-logger.conf, 添加如下内容

#Name the components on this agent
a1.sources = r1                                      # 为a1的Source组件命名为r1，多个组件用空格间隔
a1.sinks = k1                                        # 为a1的Sink组件命名为k1，多个组件用空格间隔
a1.channels = c1                                    # 为a1的Channel组件命名为c1，多个组件用空格间隔

# Describe/configure the source
a1.sources.r1.type = netcat                      # 配置r1的类型
a1.sources.r1.bind = localhost                  # 配置r1的绑定地址（注意localhost和hadoop102的区别）
a1.sources.r1.port = 44444                       # 配置r1的监听端口

# Describe the sink
a1.sinks.k1.type = logger                        # 配置k1的类型为logger,输出到控制台

# Use a channel which buffers events in memory
a1.channels.c1.type = memory                    # 配置c1的类型为memory
a1.channels.c1.capacity = 1000                 # 配置c1的容量为1000个事件
a1.channels.c1.transactionCapacity = 100     # 配置c1的事务容量为100个事件

# Bind the source and sink to the channel
a1.sources.r1.channels = c1                    # 配置r1的channel属性，指定r1连接到那个channel
a1.sinks.k1.channel = c1                        # 配置k1的channel属性，指定k1连接到那个channel

注：

• 配置文件真实使用时后面的注释、空格都要去掉，否则出错（多个组件使用空格隔开的）
• 配置文件来源于官方手册Flume 1.11.0 User Guide — Apache Flume

3.3 部署运行flume监听端口

第一种写法：

[wyr@hadoop102 flume]$ 

bin/flume-ng agent --conf conf/ --name a1 --conf-file job/simpleCase/flume-1-netcat-logger.conf -Dflume.root.logger=INFO,console

第二种写法：

[wyr@hadoop102 flume]$ 

bin/flume-ng agent -c conf/ -n a1 -f job/simpleCase/flume-1-netcat-logger.conf -Dflume.root.logger=INFO,console

[wyr@hadoop102 flume]$ bin/flume-ng agent -c conf/ -n a1 -f job/simpleCase/flume-1-netcat-logger.conf -Dflume.root.logger=INFO,console
Info: Including Hadoop libraries found via (/opt/module/hadoop-3.1.3/bin/hadoop) for HDFS access
Info: Including Hive libraries found via (/opt/module/hive) for Hive access
+ exec /opt/module/jdk1.8.0_212/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/opt/module/flume/conf:/opt/module/flume/lib/*:/opt/module/hadoop-3.1.3/etc/hadoop:/opt/module/hadoop-3.1.3/share/hadoop/common/lib/*:/opt/module/hadoop-3.1.3/share/hadoop/common/*:/opt/module/hadoop-3.1.3/share/hadoop/hdfs:/opt/module/hadoop-3.1.3/share/hadoop/hdfs/lib/*:/opt/module/hadoop-3.1.3/share/hadoop/hdfs/*:/opt/module/hadoop-3.1.3/share/hadoop/mapreduce/lib/*:/opt/module/hadoop-3.1.3/share/hadoop/mapreduce/*:/opt/module/hadoop-3.1.3/share/hadoop/yarn:/opt/module/hadoop-3.1.3/share/hadoop/yarn/lib/*:/opt/module/hadoop-3.1.3/share/hadoop/yarn/*:/opt/module/hive/lib/*' -Djava.library.path=:/opt/module/hadoop-3.1.3/lib/native org.apache.flume.node.Application -n a1 -f job/simpleCase/fl