flume在windows10上运行

1. 直接下载官网flume1.8，解压到本地F盘

2. 在conf目录下复制flume-env.ps1.template改为flume-env.ps1，改下FLUME_CLASSPATH   

$FLUME_CLASSPATH="F:\kafka\apache-flume-1.8.0-bin\lib"

  flume-env.sh操作同上

3.检查安装是否成功：  bin\flume-ng.cmd version

  在查看version时，一直遇到如下错误：

F:\kafka\apache-flume-1.8.0-bin>bin\flume-ng.cmd version

F:\kafka\apache-flume-1.8.0-bin>powershell.exe -NoProfile -InputFormat none -ExecutionPolicy unrestricted -File F:\kafka\apache-flume-1.8.0-bin\bin\flume-ng.ps1 version

WARN: Config directory not set. Defaulting to F:\kafka\apache-flume-1.8.0-bin\conf
Sourcing environment configuration script F:\kafka\apache-flume-1.8.0-bin\conf\flume-env.ps1
Test-Path : 路径中具有非法字符。
所在位置 F:\kafka\apache-flume-1.8.0-bin\bin\flume-ng.ps1:106 字符: 56
+ ...                               ? { "$_" -ne "" -and (Test-Path $_ )} |
+                                                         ~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (F:\kafka\apache-flume-1.8.0-bin\":String) [Test-Path]，ArgumentExceptio
    n
    + FullyQualifiedErrorId : ItemExistsArgumentError,Microsoft.PowerShell.Commands.TestPathCommand

conf/flume-env.ps1下的Test-Path字符非法，在该文件中ctrl+F，发现Test-Path共出现在三个地方：GetHadoopHome、GetHbaseHome、GetHiveHome。这可能与本机没有安装这三个服务有关吧，尝试注释掉这些代码行（360-405），再次运行后发现flume可以正常显示version了。如下：

F:\kafka\apache-flume-1.8.0-bin>bin\flume-ng.cmd version

F:\kafka\apache-flume-1.8.0-bin>powershell.exe -NoProfile -InputFormat none -ExecutionPolicy unrestricted -File F:\kafka\apache-flume-1.8.0-bin\bin\flume-ng.ps1 version

WARN: Config directory not set. Defaulting to F:\kafka\apache-flume-1.8.0-bin\conf
Sourcing environment configuration script F:\kafka\apache-flume-1.8.0-bin\conf\flume-env.ps1

  Running FLUME version :
    class: org.apache.flume.tools.VersionInfo
    arguments:

Flume 1.8.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 99f591994468633fc6f8701c5fc53e0214b6da4f
Compiled by denes on Fri Sep 15 14:58:00 CEST 2017
From source with checksum fbb44c8c8fb63a49be0a59e27316833d

 

4. 运行flume 案例

  不加-Dflume.root.logger=INFO,console时可正常运行，但看不到输出的序列号，加后报如下错误：

F:\kafka\apache-flume-1.8.0-bin>bin\flume-ng.cmd agent -n agent -c conf -f conf\flume-conf.properties.template  -Dflume.root.logger=INFO,console

F:\kafka\apache-flume-1.8.0-bin>powershell.exe -NoProfile -InputFormat none -ExecutionPolicy unrestricted -File F:\kafka\apache-flume-1.8.0-bin\bin\flume-ng.ps1 agent -n agent -c conf -f conf\flume-conf.properties.template  -Dflume.root.logger=INFO,console
F:\kafka\apache-flume-1.8.0-bin\bin\flume-ng.ps1 : 找不到与参数名称“Dflume.root.logger=INFO,console”匹配的参数。
    + CategoryInfo          : InvalidArgument: (:) [flume-ng.ps1]，ParentContainsErrorRecordException
    + FullyQualifiedErrorId : NamedParameterNotFound,flume-ng.ps1

  原来，在Windows上不能用Dflume这种设置(https://stackoverflow.com/questions/35067940/any-idea-about-how-to-install-flume-with-hadoop-in-windows)，应改为：

bin\flume-ng.cmd agent -n agent -c conf -f conf\flume-conf.properties.template -property  "flume.root.logger=INFO,console"

  运行后，可看到不断的序列号输出：