Goal:
Use Flume to pool a folder on local file system and write it to HDFS.
Version Information:
hadoop-0.22.0
apache-flume-1.3.1
Flume Configuration:
Edit file flume-env.sh under /$FLUME_HOME$/conf:
export JAVA_HOME=your jave home
export FLUME_CLASSPATH=your flume home
export HADOOP_CLASSPATH=your hadoop home
Edit file flume-conf.properties under /$FLUME_HOME$/conf:
# Configure the agent
agent.sources = spooldirSource
agent.channels = memoryChannel
agent.sinks = hdfsSink
# Configure the source
agent.sources.spooldirSource.type = spooldir
agent.sources.spooldirSource.spoolDir = /tmp/flume/
agent.sources.spooldirSource.channels = memoryChannel
# Configure the sink
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.hdfs.path = hdfs://masternode:9000/flume/events
agent.sinks.hdfsSink.hdfs.filePrefix = events-
agent.sinks.hdfsSink.channel = memoryChannel
# Configure the channel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100
Copy Hadoop Jars to Flume lib directory:
Copy hadoop-hdfs-0.22.0.jar and hadoop-common-0.22.0.jar to /$FLUME_HOME$/lib.
Start Flume Agent:
./bin/flume-ng agent -n agent -c conf -f conf/flume-conf.properties
Write File:
echo "Hello World">>/tmp/flume/test
View Logs:
Under /$FLUME_HOME$/logs