1、官网下载 flume 包
https://flume.apache.org/download.html
当前最新版本为:1.9
2、解压
3、修改 flume/bin/flume-ng.ps1,增加
$JAVA_OPTS="-Xmx100m -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y"
4、编写程序,自定义 Source
public class TestLogSource extends AbstractSource implements PollableSource, Configurable {
// todo:
}
5、在 flume 目录下创建如下文件夹
flume/plugins.d
flume/plugins.d/test
flume/plugins.d/test/lib
flume/plugins.d/test/extlib
flume/plugins.d/test/native
这个是官方推荐的,与 flume 自带的 jar 分开。flume 会到该目录下去查找对应的 jar 和 class。
6、将程序打包,复制 jar 到 flume/plugins.d/test/lib 目录下
7、编写 flume conf 配置文件 af.conf,指定 source、channel 和 sink
agent1.channels = c1
agent1.sources = r1
agent1.sinks = k1
#set sources
agent1.sources.r1.channels = c1
agent1.sources.r1.type = com.ka.bg.adstat.log_source.TestLogSource
agent1.sources.r1.dbinfo-file = D:/test/flume/db/aflog.db.cfg
agent1.sources.r1.ipdata-dir = D:/test/flume/ip
agent1.sources.r1.gameid-cfg = D:/test/flume/game_id.cfg
#set channels
agent1.channels.c1.type=file
agent1.channels.c1.checkpointDir=D:/test/flume/checkpoint/log-install-agent/
agent1.channels.c1.dataDirs=D:/test/flume/dataDir/log-install-agent/
# set sinks
agent1.sinks.k1.channel = c1
agent1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
agent1.sinks.k1.topic = aflog_installsignup_20190909
agent1.sinks.k1.brokerList = 10.0.2.11:9092
agent1.sinks.k1.requiredAcks = 1
agent1.sinks.k1.batchSize = 100
8、将 af.conf 放到 flume/conf 目录下
9、IDEA 开启远程调试
10、启动 flume ngent
进入 flume/bin 目录,打开 cmd 窗口,执行如下命令
flume-ng.cmd agent --conf D:/java/apache-flume-1.9.0-bin/conf --conf-file D:/java/apache-flume-1.9.0-bin/conf/af.cfg --name agent1
11、在需要调试的代码上打上断点,点击调试按钮
断点调试成功
12、查看 Kafka 队列数据
./kafka-console-consumer.sh --bootstrap-server aflog_installsignup_20190909 --topic test-online --from-beginning --group test