Flume 安装与配置步骤

1.解压

tar -zxvf apache-flume-1.9.0-bin.tar.gz

2.配置环境变量 

vim /etc/profile

export FLUME_HOME=/home/wang/soft/flume/apache-flume-1.9.0-bin
export PATH=$PATH:$FLUME_HOME/bin
source /etc/profile

3.创建必要的目录

 mkdir -p $FLUME_HOME/conf

4.创建 Flume 配置文件

cd $FLUME_HOME/conf
vim flume-kafka.conf 

5.配置文件内容 （红色部分是Nginx日志所在的路径）

# 定义组件
agent.sources = logSource
agent.channels = memoryChannel
agent.sinks = kafkaSink

# 配置 source
agent.sources.logSource.type = exec
agent.sources.logSource.command = tail -F /usr/local/nginx/logs/access.log
agent.sources.logSource.channels = memoryChannel

# 配置 channel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 1000
agent.channels.memoryChannel.transactionCapacity = 100

# 配置 sink
agent.sinks.kafkaSink.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.kafkaSink.kafka.bootstrap.servers = localhost:9092
agent.sinks.kafkaSink.kafka.topic = logs
agent.sinks.kafkaSink.kafka.flumeBatchSize = 20
agent.sinks.kafkaSink.channel = memoryChannel

6.检查 Flume 的 lib 目录

ls -l $FLUME_HOME/lib 

7. 启动 Flume 

cd $FLUME_HOME
$FLUME_HOME/bin/flume-ng agent -n agent -c conf -f conf/flume-kafka.conf

 8.验证 Flume 是否正常运行

jps

应该能看到 Flume 相关进程。 

9.检查日志 

tail -f $FLUME_HOME/logs/flume.log