Flume【问题记录 01】【at org.apache.flume.node.Application.main(Application.java:xxx) 类问题整理+其他类型问题总结】【避坑指南】

我使用的是 apache-flume-1.9.0 版本，以下问题和解决均以此版本进行说明。

[root@tcloud ~]# flume-ng version
Flume 1.9.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: d4fcab4f501d41597bc616921329a4339f73585e
Compiled by fszabo on Mon Dec 17 20:45:25 CET 2018
From source with checksum 35db629a3bda49d23e9b3690c80737f9

test-exec-memory-logger.properties配置文件内容如下【备注信息不再赘述】：

a1.sources = s1 
a1.channels = c1 
a1.sinks = k1 

a1.sources.s1.type = exec
a1.sources.s1.command = tail -F /tmp/log.txt
a1.sources.s1.shell = /bin/bash -c
a1.channels.c1.type = memory
a1.sinks.k1.type = logger

a1.sources.s1.channels = c1 
a1.sinks.k1.channel = c1 

1.问题

# 执行的命令
[root@tcloud ~]# flume-ng agent \
--conf conf--conf-file /home/flume/test/test-exec-memory-logger.properties \
--name a1-Dflume.root.logger=INFO,console

# 报错信息
2021-08-24 13:39:20,589 ERROR node.Application:
 A fatal error occurred while running. Exception follows.
java.lang.NullPointerException
        at java.io.File.<init>(File.java:277)
        at org.apache.flume.node.Application.main(Application.java:325)

2.原因

我去官网查看了启动命令：

Starting an agent : An agent is started using a shell script called flume-ng which is located in the bin directory of the Flume distribution. You need to specify the agent name, the config directory, and the config file on the command line:

$ bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template

Now the agent will start running source and sinks configured in the given properties file.

随后我把双横杠 – 改为了单横杠 - 又试了以下：

[root@tcloud ~]# flume-ng agent \
-n a1 \
-c conf \
-f /home/flume/test/test-exec-memory-logger.properties \
-Dflume.root.logger=INFO,console

启动成功了，我以为是我的启动参数有问题，其实并不是，我发现我的参数写的不对，少了空格：

[root@tcloud ~]# flume-ng agent \
--conf conf \
--conf-file /home/flume/test/test-exec-memory-logger.properties \
--name a1 \
-Dflume.root.logger=INFO,console

此时正常启动了 😅 尴尬，这是不认真导致的问题。

3.延申

我准备测试以下启动时会报的错。

3.1 agentName错误

配置文件test-exec-memory-logger.properties里配置的agentName是a1这里【故意写错成a2】：

[root@tcloud ~]# flume-ng agent \
--conf conf \
--conf-file /home/flume/test/test-exec-memory-logger.properties \
--name a2 \
-Dflume.root.logger=INFO,console

2021-08-24 13:59:49,549 INFO node.PollingPropertiesFileConfigurationProvider:
Configuration provider starting
Reloading configuration file:/home/flume/test/test-exec-memory-logger.properties
2021-08-24 13:59:49,558 INFO conf.FlumeConfiguration: 
Processing:s1
Added sinks: k1 Agent: a1
Processing:c1
Processing:k1
2021-08-24 13:59:49,559 WARN conf.FlumeConfiguration: 
Agent configuration for 'a1' has no configfilters.
Post-validation flume configuration contains configuration for agents: [a1]
2021-08-24 13:59:49,599 WARN node.AbstractConfigurationProvider: 
No configuration found for this host:a2
2021-08-24 13:59:49,604 INFO node.Application:
Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }

3.2 配置文件不存在

[root@tcloud ~]# flume-ng agent \
--conf conf \
--conf-file /home/flume/test/test2-exec-memory-logger.properties \
--name a1 \
-Dflume.root.logger=INFO,console

2021-08-24 14:05:42,889 ERROR node.Application: 
A fatal error occurred while running. Exception follows.
org.apache.commons.cli.ParseException: 
The specified configuration file does not exist: 
/home/flume/test/test2-exec-memory-logger.properties
at org.apache.flume.node.Application.main(Application.java:342)

3.3 配置参数名错误

将配置参数–name故意写错成【–naem】：

[root@tcloud ~]# flume-ng agent \
--conf conf \
--conf-file /home/flume/test/test2-exec-memory-logger.properties \
--naem a1 \
-Dflume.root.logger=INFO,console

2021-08-24 14:22:25,174 ERROR node.Application: A fatal error occurred while running. Exception follows.org.apache.commons.cli.UnrecognizedOptionException: 
Unrecognized option: --naem
        at org.apache.commons.cli.Parser.processOption(Parser.java:363)
        at org.apache.commons.cli.Parser.parse(Parser.java:199)
        at org.apache.commons.cli.Parser.parse(Parser.java:85)
        at org.apache.flume.node.Application.main(Application.java:287)

3.4 配置文件内容有误

test-exec-memory-logger.properties配置文件故意将c1【写错成c2】：

a1.sources = s1 
a1.channels = c2 
a1.sinks = k1 

a1.sources.s1.type = exec
a1.sources.s1.command = tail -F /tmp/log.txt
a1.sources.s1.shell = /bin/bash -c
a1.channels.c1.type = memory
a1.sinks.k1.type = logger

a1.sources.s1.channels = c1 
a1.sinks.k1.channel = c1 

报错信息【does not contain any valid channels】是说a1没有channels【实质上是c2未进行配置】：

2021-08-24 14:13:37,571 WARN conf.FlumeConfiguration: 
Agent configuration for 'a1' has no configfilters.
Agent configuration for 'a1' does not contain any valid channels. Marking it as invalid.
Agent configuration invalid for agent 'a1'. It will be removed.
Post-validation flume configuration contains configuration for agents: []
2021-08-24 14:13:37,596 WARN node.AbstractConfigurationProvider: 
No configuration found for this host:a1

test-exec-memory-logger.properties配置文件【channels的type故意写错成brain】：

a1.sources = s1 
a1.channels = c1
a1.sinks = k1 

a1.sources.s1.type = exec
a1.sources.s1.command = tail -F /tmp/log.txt
a1.sources.s1.shell = /bin/bash -c
a1.channels.c1.type = brain
a1.sinks.k1.type = logger

a1.sources.s1.channels = c1 
a1.sinks.k1.channel = c1 

报错信息【Failed to load configuration data】配置信息无法加载，说明是配置信息有误：

2021-08-24 14:18:21,408 ERROR node.PollingPropertiesFileConfigurationProvider: 
Failed to load configuration data. 
Exception follows.
org.apache.flume.FlumeException: Unable to load channel type: brain, class: brain