本地 IDEA 对 Flume 进行远程调试

1、官网下载 flume 包

https://flume.apache.org/download.html

当前最新版本为：1.9

2、解压

3、修改 flume/bin/flume-ng.ps1，增加

$JAVA_OPTS="-Xmx100m -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y"

4、编写程序，自定义 Source

public class TestLogSource extends AbstractSource implements PollableSource, Configurable {
  // todo:
}

5、在 flume 目录下创建如下文件夹

flume/plugins.d

flume/plugins.d/test

flume/plugins.d/test/lib

flume/plugins.d/test/extlib

flume/plugins.d/test/native

这个是官方推荐的，与 flume 自带的 jar 分开。flume 会到该目录下去查找对应的 jar 和 class。

6、将程序打包，复制 jar 到 flume/plugins.d/test/lib 目录下

7、编写 flume conf 配置文件 af.conf，指定 source、channel 和 sink

agent1.channels = c1
agent1.sources = r1
agent1.sinks = k1

#set sources
agent1.sources.r1.channels = c1
agent1.sources.r1.type = com.ka.bg.adstat.log_source.TestLogSource
agent1.sources.r1.dbinfo-file = D:/test/flume/db/aflog.db.cfg
agent1.sources.r1.ipdata-dir = D:/test/flume/ip
agent1.sources.r1.gameid-cfg = D:/test/flume/game_id.cfg

#set channels
agent1.channels.c1.type=file
agent1.channels.c1.checkpointDir=D:/test/flume/checkpoint/log-install-agent/
agent1.channels.c1.dataDirs=D:/test/flume/dataDir/log-install-agent/

# set sinks
agent1.sinks.k1.channel = c1
agent1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
agent1.sinks.k1.topic = aflog_installsignup_20190909
agent1.sinks.k1.brokerList = 10.0.2.11:9092
agent1.sinks.k1.requiredAcks = 1
agent1.sinks.k1.batchSize = 100

8、将 af.conf 放到 flume/conf 目录下

9、IDEA 开启远程调试

10、启动 flume ngent

进入 flume/bin 目录，打开 cmd 窗口，执行如下命令

flume-ng.cmd agent --conf D:/java/apache-flume-1.9.0-bin/conf --conf-file D:/java/apache-flume-1.9.0-bin/conf/af.cfg --name agent1

11、在需要调试的代码上打上断点，点击调试按钮

 断点调试成功

12、查看 Kafka 队列数据

./kafka-console-consumer.sh --bootstrap-server aflog_installsignup_20190909 --topic test-online --from-beginning --group test