flume流程之http-memory-hdfs

主要需要了解http source的一些属性,最重要的就是bind以及监听端口.
配置参数 默认值 描述

# 命名个组件
b1001.sources = r1
b1001.sinks = k1
b1001.channels = c1
 
#source属性
b1001.sources.r1.type = http
b1001.sources.r1.bind=0.0.0.0
b1001.sources.r1.port=9666

 
# 描述channel属性
b1001.channels.c1.type = memory
b1001.channels.c1.capacity = 1000
b1001.channels.c1.transactionCapacity = 100

# 描述sink属性
b1001.sinks.k1.type=hdfs
b1001.sinks.k1.hdfs.path=hdfs://mypc01:8020/news-article1/%Y%m%d
b1001.sinks.k1.hdfs.filePrefix=news-
b1001.sinks.k1.hdfs.fileSuffix = .txt
b1001.sinks.k1.hdfs.rollSize=102400
b1001.sinks.k1.hdfs.rollCount	= 0
#单位为s
b1001.sinks.k1.hdfs.rollInterval=60
b1001.sinks.k1.hdfs.useLocalTimeStamp = true
 

 
# 关联source和sink到channel
b1001.sources.r1.channels = c1
b1001.sinks.k1.channel = c1

启动脚本

flume-ng启动脚本. -c指定flume自身的一些配置.比如java位置什么的
.-f指定agent的配置.主要是自己写的flume方案.
这两个参数不要搞混淆了

#bin/bash
/usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf \
-f /opt/apps/collect-app/scripts/collect-app-http-flume.conf \
-n b1001 -Dflume.root.logger=INFO,console -Dflume.monitoring.type=http -Dflume.monitoring.port=31002

测试

向mypc01 的9666端口发送信息,这样flume上配置的source就可以监听到.此外,默认情况下,http source只能处理json格式的数据. 必须首先启动flume-ng 才可以测试.负责会是拒绝连接.

curl -X POST http://mypc01:9666  -d  '[{"header":{"name":"article"},"body":"123"}]'

ps:

flume-ng命令官方帮助如下

global options:
  --conf,-c <conf>          use configs in <conf> directory
  --classpath,-C <cp>       append to the classpath
  --dryrun,-d               do not actually start Flume, just print the command
  --plugins-path <dirs>     colon-separated list of plugins.d directories. See the
                            plugins.d section in the user guide for more details.
                            Default: $FLUME_HOME/plugins.d
  -Dproperty=value          sets a Java system property value
  -Xproperty=value          sets a Java -X option

agent options:
  --name,-n <name>          the name of this agent (required)
  --conf-file,-f <file>     specify a config file (required if -z missing)
  --zkConnString,-z <str>   specify the ZooKeeper connection to use (required if -f missing)
  --zkBasePath,-p <path>    specify the base path in ZooKeeper for agent configs
  --no-reload-conf          do not reload config file if changed
  --help,-h                 display help text