How to setup FlumeNG with example configuration

1. Check out the source

For those that prefer subversion:

$ svn checkout https://svn.apache.org/repos/asf/incubator/flume/trunk/

If you're more of a git person:

$ git clone git://git.apache.org/flume.git
$ cd flume
$ git checkout trunk

Note: The git repo is a read-only mirror of the subversion repo.

2. Compile the project

# Build the code and run the tests
$ mvn package
# ...or build the code without running the tests
$ mvn package -DskipTests

This produces two types of packages in flume-ng-dist/target. They are:

• flume-ng-dist-1.2.0-incubating-SNAPSHOT-dist.tar.gz - A binary distribution of Flume, ready to run.
• flume-ng-dist-1.2.0-incubating-SNAPSHOT-src.tar.gz - A source-only distribution of Flume.

If you're a user and you just want to run Flume, you probably want the -dist version. Copy one out, decompress it, and you're ready to go.

$ cp flume-ng-dist/target/flume-ng-dist-1.2.0-incubating-SNAPSHOT-dist.tar.gz .
$ tar -zxvf flume-ng-dist-1.2.0-incubating-SNAPSHOT-dist.tar.gz
$ cd flume-1.2.0-incubating-SNAPSHOT

3. Create your own properties file based on the working template (or create one from scratch)

$ cp conf/flume-conf.properties.template conf/flume.conf

4. (Optional) Create your flume-env.sh file based on the template (or create one from scratch). The flume-ng executable looks for and sources a file named "flume-env.sh" in the conf directory specified by the --conf/-c commandline option. One use case for using flume-env.sh would be to specify debugging or profiling options via JAVA_OPTS when developing your own custom Flume NG components such as sources and sinks.

$ cp conf/flume-env.sh.template conf/flume-env.sh

5. Configure 

FlumeNG configure is format is <identifier>.type.subtype.parameter.config, where <identifier> is the name of the agent we call later to startup.  Take below conf file for example, 

<identifier> --> syslog-agent 

type --> sources, sinks, channels

subtype --> Syslog, HDFS-LAB, MemoryChannel-1

parameter --> type, port, channels, channel(so strange for different), hdfs

config --> path, file.Prefix, 

syslog-agent.sources = Syslog
syslog-agent.channels = MemoryChannel-1
syslog-agent.sinks = HDFS-LAB

syslog-agent.sources.Syslog.type = syslogTcp
syslog-agent.sources.Syslog.port = 5140
syslog-agent.sources.Syslog.channels = MemoryChannel-1

syslog-agent.sinks.HDFS-LAB.channel = MemoryChannel-1
syslog-agent.sinks.HDFS-LAB.type = hdfs
syslog-agent.sinks.HDFS-LAB.hdfs.path = hdfs://NN.URI:PORT/flumetest/'%{host}''
syslog-agent.sinks.HDFS-LAB.hdfs.file.Prefix = syslogfiles
syslog-agent.sinks.HDFS-LAB.hdfs.file.rollInterval = 60
syslog-agent.sinks.HDFS-LAB.hdfs.file.Type = SequenceFile

syslog-agent.channels.MemoryChannel-1.type = memory

sample config run on psedule hadoop:

agent.channels = c1
agent.sources = r1
agent.sinks = k1
#
agent.channels.c1.type = MEMORY
#
agent.sources.r1.channels = c1
agent.sources.r1.type = SEQ
#
agent.sinks.k1.channel = c1
agent.sinks.k1.type = LOGGER
#
agent.sinks.k1.channel = c1
agent.sinks.k1.type = HDFS
agent.sinks.k1.hdfs.path = hdfs://localhost/tmp/data/flume 
#hdfs://<host>:<port>:<path>
agent.sinks.k1.hdfs.fileType = DataStream
agent.sinks.k1.hdfs.codeC = SnappyCodec

6. Run Flume NG

After you've configured Flume NG (see below), you can run it with the bin/flume-ng executable. This script has a number of arguments and modes.

Issue a flume command to start agent2(Which puts data on HDFS).

              flume-ng node -c ../conf/ -f ../conf/agent2.properties -n agent2

-c define global configuration folder.

-f The configuration filename for the Flume-NG node(Mandatory)

-n The name of the Flume-NG node(Mandatory)

This would start a node with HDFS as a sink,Apache Avro as a source and memory as a channel.

Start the flow

Flume-ng starts a single flow per process. That's will be done with:

bin/flume-ng agent -n YOUR_IDENTIFIER -f YOUR_CONFIGFILE 

eg:

$bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf

$ bin/flume-ng agent --conf conf/ -f conf/flume.conf -n agent1

$ flume-ng agent -n agent -f /usr/lib/flume-ng/conf/flume-conf.properties

For reference:

flume-ng global options

Option	Description
--conf,-c <conf>	Use configs in <conf> directory
--classpath,-C <cp>	Append to the classpath
--dryrun,-d	Do not actually start Flume, just print the command
-Dproperty=value	Sets a JDK system property value

flume-ng agent options

When given the agent command, a Flume NG agent will be started with a given configuration file (required).

Option	Description
--conf-file,-f <file>	Indicates which configuration file you want to run with (required)
--name,-n <agentname>	Indicates the name of agent on which we're running (required)