Flume向HDFS写数据实例

最新推荐文章于 2022-03-03 08:21:11 发布

weixin_33670786

最新推荐文章于 2022-03-03 08:21:11 发布

阅读量131

点赞数

文章标签：大数据 python

原文链接：https://my.oschina.net/xiangchen/blog/101111

版权

2019独角兽企业重金招聘Python工程师标准>>>

Goal:

Use Flume to pool a folder on local file system and write it to HDFS.

Version Information:

hadoop-0.22.0

apache-flume-1.3.1

Flume Configuration:

Edit file flume-env.sh under /$FLUME_HOME$/conf:

export JAVA_HOME=your jave home 
export FLUME_CLASSPATH=your flume home
export HADOOP_CLASSPATH=your hadoop home

Edit file flume-conf.properties under /$FLUME_HOME$/conf:

# Configure the agent
agent.sources = spooldirSource
agent.channels = memoryChannel
agent.sinks = hdfsSink

# Configure the source
agent.sources.spooldirSource.type = spooldir
agent.sources.spooldirSource.spoolDir = /tmp/flume/
agent.sources.spooldirSource.channels = memoryChannel

# Configure the sink
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.hdfs.path = hdfs://masternode:9000/flume/events
agent.sinks.hdfsSink.hdfs.filePrefix = events-
agent.sinks.hdfsSink.channel = memoryChannel

# Configure the channel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100

Copy Hadoop Jars to Flume lib directory:

Copy hadoop-hdfs-0.22.0.jar and hadoop-common-0.22.0.jar to /$FLUME_HOME$/lib.

Start Flume Agent:

./bin/flume-ng agent -n agent -c conf -f conf/flume-conf.properties

Write File:

echo "Hello World">>/tmp/flume/test

View Logs:

Under /$FLUME_HOME$/logs

转载于:https://my.oschina.net/xiangchen/blog/101111