(1) HelloWorld
Starting an agent
An agent is started using a shell script called flume-ng which is located in the bin directory of the Flume distribution. You need to specify the agent name, the config directory, and the config file on the command line:
Now the agent will start running source and sinks configured in the given properties file.
A simple example
Here, we give an example configuration file, describing a single-node Flume deployment. This configuration lets a user generate events and subsequently logs them to the console.
This configuration defines a single agent named a1. a1 has a source that listens for data on port 44444, a channel that buffers event data in memory, and a sink that logs event data to the console. The configuration file names the various components, then describes their types and configuration parameters. A given configuration file might define several named agents; when a given Flume process is launched a flag is passed telling it which named agent to manifest.
Given this configuration file, we can start Flume as follows:
Note that in a full deployment we would typically include one more option: --conf=<conf-dir>. The <conf-dir> directory would include a shell script flume-env.sh and potentially a log4j properties file. In this example, we pass a Java option to force Flume to log to the console and we go without a custom environment script.
From a separate terminal, we can then telnet port 44444 and send Flume an event:
The original Flume terminal will output the event in a log message.
Congratulations - you’ve successfully configured and deployed a Flume agent! Subsequent sections cover agent configuration in much more detail.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(2)单节点 Flume 直接写入 HDFS
查阅文章:http://my.oschina.net/leejun2005/blog/288136#OSC_h2_10 按照3.2章节练习(flume版本1.5.2)
遇到问题:
▂ ▃ ▄ ▅ ▆ ▇ █
2014-12-15 17:22:28,418 (conf-file-poller-0) [ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)] Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType
at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:251)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:418)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 12 more
意思就是:在类路径下找不到什么什么calss.....应该是缺少jar包了 。网上折腾了半天,官网的说明也看了。
单节点flume写入HDFS中 用到的sink是 HDFS Sink。
This sink writes events into the Hadoop Distributed File System (HDFS). It currently supports creating text and sequence files. It supports compression in both file types. The files can be rolled (close current file and create a new one) periodically based on the elapsed time or size of data or number of events. It also buckets/partitions data by attributes like timestamp or machine where the event originated. The HDFS directory path may contain formatting escape sequences that will replaced by the HDFS sink to generate a directory/file name to store the events. Using this sink requires hadoop to be installed so that Flume can use the Hadoop jars to communicate with the HDFS cluster. Note that a version of Hadoop that supports the sync() call is required.
大意就是:使用这种sink需要hadoop的jar包。由于也没说具体哪个jar包
(1)org/apache/hadoop/io/SequenceFile$CompressionType 是在hadoop-2.3.0-cdh5.1.0\share\hadoop\common\hadoop-common-2.3.0-cdh5.1.0.jar中 , 将该jar包放到flume的lib包中。再起动还是报同样的错。
(2)查看env (服务器装了两个flume A配置了FLUME_HOME,B我练习的没有配置),于是进到conf中修改flume-env.sh
JAVA_HOME=/home/xxx/jdk/jdk1.7.0_55(原来已有)
FLUME_HOME=/home/xxx/pjmflume/flume(此处添加)
(3)再起报错 Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration
缺少了 commons-configuration-1.6.jar 包。添上。
(4)再起报错 Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
缺少 hadoop-auth-2.3.0-cdh5.1.0.jar 包。添上。
(5)报错
2014-12-15 17:42:34,356 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:463)] HDFS IO error
java.io.IOException: No FileSystem for scheme: hdfs
缺少 hadoop-hdfs-2.3.0-cdh5.1.0.jar 包。添上
----------------------------------------------------------------------------
理论上 就是flume的 hdfs sink需要用到hadoop的上述jar包。 懒得一个一个错误的排查可直接将hadoop share下的所有jar包 拷贝一份移至
flume的lib中即可。
----------------------------------------------------------------------------
Caused by: java.net.NoRouteToHostException: No route to host
保通信错误: 猜测应该是 本机 与 hdfs集群不在一个网络上。
telnet ip及端口信息
telnet 10.211.151.4 9111
Trying 10.211.151.4...
telnet: connect to address 10.211.151.4: No route to host
telnet: Unable to connect to remote host: No route to host
最后换了台可以hdfs交互的服务器练习。可以成功。
-----------------------------------------------------------------------------------------------------------------------