Pig flume流配置文件

首先分享一个网站，个人觉得这篇写Flume NG 简介及配置实战的文章非常好，下面也有好多案例可以实践操作

https://my.oschina.net/leejun2005/blog/288136

1)将下载的flume包上传到/usr/local，解压到/usr/local目录中
tar –zxvf apache-flume-1.6.0-bin.tar.gz -C /usr/local
ls后看到apache-flume-1.6.0-bin


2)修改 flume-env.sh 配置文件,主要是JAVA_HOME变量设置
[root@master apache-flume-1.6.0-bin]# cp conf/flume-env.sh.template conf/flume-env.sh
[root@master apache-flume-1.6.0-bin]# vi conf/flume-env.sh
 在文件中添加路径    


 JAVA_HOME：/usr/java/jdk1.7.0_80


3）vim /etc/profile


添加flume路径
xport FLUME_PATH=/usr/local/apache-flume-1.6.0-bin
修改
export PATH=$HIVE_HOME/bin:$PIG_HOME/bin:$FLUME_PATH/bin:$PATH
 
4）resource /etc/profile


5）[root@master bin]# flume-ng version
出现下面的内容即成功
Flume 1.6.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 2561a23240a71ba20bf288c7c2cda88f443c2080
Compiled by hshreedharan on Mon May 11 11:15:44 PDT 2015
From source with checksum b29e416802ce9ece3269d34233baf43f




1)案例1：Avro
　　　　
    Avro可以发送一个给定的文件给Flume，Avro 源使用AVRO RPC机制。
　　
  a)创建agent配置文件
    vi /usr/local/apache-flume-1.6.0-bin/conf/avro.conf
内容：
a1.sources = r1
a1.sinks = k1
a1.channels = c1
  
# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.channels = c1
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 4141
  
# Describe the sink
a1.sinks.k1.type = logger
  
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
  
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1


  b)启动flume agent a1
/usr/local/apache-flume-1.6.0-bin/bin/flume-ng agent -c /usr/local/apache-flume-1.6.0-bin/conf -f /usr/local/apache-flume-1.6.0-bin/conf/avro.conf -n a1 -Dflume.root.logger=INFO,console


　c)创建指定文件
echo "hello world" >> /usr/local/apache-flume-1.6.0-bin/log.00


  d)使用avro-client发送文件
/usr/local/apache-flume-1.6.0-bin/bin/flume-ng avro-client -c . -H 0.0.0.0 -p 4141 -F /usr/local/apache-flume-1.6.0-bin/log.00


　f)在m1的控制台，可以看到以下信息，注意最后一行：


-08-10 10:43:26,718 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64        hello world }