flume安装与实战

flume安装：

官网下载:

http://archive.apache.org/dist/flume/

一.进入该下载包目录:

cd  /tools

wget http://archive.apache.org/dist/flume/1.8.0/apache-flume-1.8.0-bin.tar.gz

tar -zxvf apache-flume-1.8.0-bin.tar.gz  -C /usr/local/

chown -R hadoop:hadoop apache-flume-1.8.0-bin/

vi /etc/profile

# flume config

export FLUME_HOME=/usr/local/apache-flume-1.8.0-bin

export PATH=$PATH:$FLUME_HOME/bin

 

source /etc/profile

 

在任意目录执行flume-ng命令,若能成功输出下面的信息,说明环境变量配置成功

commands:

  help                      display this help text

  agent                     run a Flume agent

  avro-client               run an avro Flume client

  version                   show Flume version info

 

vi flume-conf.properties

# 单节点Flume配置例子

# 给Agent 中的三个组件Source  Sink 和 Channel 各取一个别名,a1代表为Agent取的别名

a1.sources = r1

a1.channels = c1

a1.sinks = k1

 

# 给 Source 属性配置

a1.sources.r1.type = netcat

a1.sources.r1.bind = localhost

a1.sources.r1.port = 44444

 

# 绑定source 和 sink 到 channel 上

a1.sinks.k1.channel = c1

a1.sources.r1.channels = c1

 

# sink 属性配置信息

a1.sinks.k1.type = logger

# Each channel's type is defined.

 

# channel 属性配置信息

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

 

场景一:

flume-ng agent --conf conf --conf-file /usr/local/apache-flume-1.8.0-bin/conf/flume-conf.properties --name a1 -Dflume.root.logger=INFO,console 

新开一个连接node-1的SSH窗口

telnet localhost 44444

连接成功后: 随便输入内容,后台启动agent的SSH窗口

 

场景二: 日志监控(1)

 

vi flume-conf-01.properties

# 单节点Flume配置例子

# 给Agent 中的三个组件Source  Sink 和 Channel 各取一个别名,a1代表为Agent取的别名

a1.sources = r1

a1.channels = c1

a1.sinks = k1

 

# 给 Source 属性配置

a1.sources.r1.type = exec

a1.sources.r1.command = tail -F /home/hadoop/data.log

# 绑定source 和 sink 到 channel 上

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

 

# sink 属性配置信息

a1.sinks.k1.type = logger

 

# channel 属性配置信息

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

 

flume-ng agent --name a1 --conf conf --conf-file /usr/local/apache-flume-1.8.0-bin/conf/flume-conf-01.properties  -Dflume.root.logger=INFO,console

 

另外一个SSH窗口

echo 'laoggaogaoo4156465413' >> /home/hadoop/data.log

 

场景三:日志监控(2)

配置文件的内容如下:(接收池为sinks = hdfs)

vi  flume-conf-hdfs.properties

 

# 给Agent 中的三个组件Source  Sink 和 Channel 各取一个别名,a1代表为Agent取的别名

a1.sources = r1

a1.channels = c1

a1.sinks = k1

 

# 给 Source 属性配置

a1.sources.r1.type = exec

a1.sources.r1.command = tail -F /usr/local/elasticsearch-6.5.4/logs/gc.log.0.current

 

# 绑定source 和 sink 到 channel 上

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

 

# 组件类型为hdfs

a1.sinks.k1.type = hdfs

a1.sinks.k1.hdfs.path = hdfs://node-1:9000/flume/%Y-%m

a1.sinks.k1.hdfs.filePrefix = %Y-%m-%d-%H

a1.sinks.k1.hdfs.fileSuffix = .log

a1.sinks.k1.hdfs.useLocalTimeStamp = true

a1.sinks.k1.hdfs.minBlockReplicas = 1

a1.sinks.k1.hdfs.fileType = DataStream

a1.sinks.k1.hdfs.writerFormat = Text

a1.sinks.k1.hdfs.rollInterval = 86400

a1.sinks.k1.hdfs.rollSize = 1000000

a1.sinks.k1.hdfs.rollCount = 10000

 

# channel 属性配置信息

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

                                                           

 

执行如下的命令:

flume-ng agent --name a1 --conf conf --conf-file /usr/local/apache-flume-1.8.0-bin/conf/flume-conf-hdfs.properties  -Dflume.root.logger=INFO,console

如果执行命令下面的错误：

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create file/flume/2019-11/2019-11-10-21.1573393841845.log.tmp. Name node is in safe mode.

Resources are low on NN. Please add or free up more resources then turn off safe mode manually. NOTE:  If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off. NamenodeHostName:node-1

 

执行该命令

hdfs dfsadmin -safemode leave

Safe mode is OFF

http://node-1:50070/  进入hadoop的web ui页面查看生成的日志

hadoop fs -ls -R /flume/              任意一个节点执行,查看flume下生成的日志文件

hadoop fs -cat /flume/2019-11/*        查看日志生成的内容

.......