Flume使用

最新推荐文章于 2018-02-01 16:30:04 发布
最新推荐文章于 2018-02-01 16:30:04 发布
阅读量196 收藏 1
点赞数
分类专栏： 大数据
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_40652340/article/details/78854731
版权
flume使用
1、source读取hive日志
2、选择mem channel
3、sink-》日志文件
4、给agent起一个名称，名称任意
5、初始化三大组件的名称
a1.sources = s1
a1.channels = c1
a1.sinks = k1
6、channel的存储是按照event来计算的
7、运行格式：

需求：source：读hive日志  channel：mem  sink：log

【memory channel】

hive-mem-log.properties文件内容如下：

# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per a1, 
# in this case called 'a1'

a1.sources = s1
a1.channels = c1
a1.sinks = k1

# For each one of the sources, the type is defined
#指定source的类型
a1.sources.s1.type = exec
#执行的命令
a1.sources.s1.command=tail -f /opt/cdh5/hive-0.13.1-cdh5.3.6/logs/hive.log
# The channel can be defined as follows.
a1.sources.s1.channels = c1

# Each sink's type must be defined
#指定sink的类型
a1.sinks.k1.type = logger

#Specify the channel the sink should use
a1.sinks.k1.channel = c1

# Each channel's type is defined.
#指定channel的类型
a1.channels.c1.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
#指定channel中最多包含几个event
a1.channels.c1.capacity = 100
a1.channels.c1.transactionCapacity = 100

执行命令：bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/hive-mem-log.properties -Dflume.root.logger=INFO,console

8、flume启动顺序：
channel-》sink-》source
9、flume关闭顺序：
source-》sink-》channel


【file channel】

hive-file-log.properties文件内容如下：

# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per a1, 
# in this case called 'a1'

a1.sources = s1
a1.channels = c1
a1.sinks = k1

# For each one of the sources, the type is defined
a1.sources.s1.type = exec
a1.sources.s1.command=tail -f /opt/cdh5/hive-0.13.1-cdh5.3.6/logs/hive.log
# The channel can be defined as follows.
a1.sources.s1.channels = c1

# Each sink's type must be defined
a1.sinks.k1.type = logger

#Specify the channel the sink should use
a1.sinks.k1.channel = c1

# Each channel's type is defined.
a1.channels.c1.type = file

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
#检查点目录
a1.channels.c1.checkpointDir = /opt/datas/flume/file/check
#数据存储目录
a1.channels.c1.dataDirs = /opt/datas/flume/file/data

执行命令：bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/hive-file-log.properties -Dflume.root.logger=INFO,console


【hdfs sink】

# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per a1, 
# in this case called 'a1'

a1.sources = s1
a1.channels = c1
a1.sinks = k1

# For each one of the sources, the type is defined
a1.sources.s1.type = exec
a1.sources.s1.command=tail -f /opt/cdh5/hive-0.13.1-cdh5.3.6/logs/hive.log
# The channel can be defined as follows.
a1.sources.s1.channels = c1

# Each sink's type must be defined
#指定sink类型
a1.sinks.k1.type = hdfs
#指定hdfs上保存日志的目录
a1.sinks.k1.hdfs.path = /flume/event/hdfs
#指定文件头名称
a1.sinks.k1.hdfs.filePrefix = hive-log
#Specify the channel the sink should use
a1.sinks.k1.channel = c1

# Each channel's type is defined.
a1.channels.c1.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 1000
执行命令：bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/hive-hdfs-mem-log.properties -Dflume.root.logger=INFO,console


【文件名称、大小、分区】

 hive-size-mem-log.properties 文件内容如下：

# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per a1, 
# in this case called 'a1'

a1.sources = s1
a1.channels = c1
a1.sinks = k1

# For each one of the sources, the type is defined
a1.sources.s1.type = exec
a1.sources.s1.command=tail -f /opt/cdh5/hive-0.13.1-cdh5.3.6/logs/hive.log
# The channel can be defined as follows.
a1.sources.s1.channels = c1

# Each sink's type must be defined
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /flume/event/size01
a1.sinks.k1.hdfs.filePrefix = hive-log
#指定一个文件的大小 10KB
a1.sinks.k1.hdfs.rollSize = 10240
#指定多长时间生成一个文件默认30秒
a1.sinks.k1.hdfs.rollInterval = 0
#指定每个文件中events的数量 默认10个event为一个文件
a1.sinks.k1.hdfs.rollCount = 0
#Specify the channel the sink should use
a1.sinks.k1.channel = c1

# Each channel's type is defined.
a1.channels.c1.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 1000

执行命令：bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/hive-size-mem-log.properties -Dflume.root.logger=INFO,console


hive-part-mem-log.properties文件内容如下：

# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per a1, 
# in this case called 'a1'

a1.sources = s1
a1.channels = c1
a1.sinks = k1

# For each one of the sources, the type is defined
a1.sources.s1.type = exec
a1.sources.s1.command=tail -f /opt/cdh5/hive-0.13.1-cdh5.3.6/logs/hive.log
# The channel can be defined as follows.
a1.sources.s1.channels = c1

# Each sink's type must be defined
a1.sinks.k1.type = hdfs
#指定分区的目录
a1.sinks.k1.hdfs.path = /flume/event/date=%Y-%m-%d
a1.sinks.k1.hdfs.filePrefix = hive-log
a1.sinks.k1.hdfs.rollSize = 10240
a1.sinks.k1.hdfs.rollInterval = 0
a1.sinks.k1.hdfs.rollCount = 0
#必须指定时间标准用于 event的head头部 指定用linux本地时间作为时间搓
a1.sinks.k1.hdfs.useLocalTimeStamp = true
#Specify the channel the sink should use
a1.sinks.k1.channel = c1

# Each channel's type is defined.
a1.channels.c1.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 1000
执行命令：bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/hive-part-mem-log.properties -Dflume.root.logger=INFO,console


【spooling dir source 】监控文件夹

# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per a1, 
# in this case called 'a1'

a1.sources = s1
a1.channels = c1
a1.sinks = k1

# For each one of the sources, the type is defined
#指定source类型
a1.sources.s1.type = spooldir
#指定要监控的文件夹
a1.sources.s1.spoolDir = /opt/datas/flume/spooling/
#不匹配.tmp结尾的文件
a1.sources.s1.ignorePattern = ([^ ]*\.tmp$)
# The channel can be defined as follows.
a1.sources.s1.channels = c1

# Each sink's type must be defined
a1.sinks.k1.type = hdfs
#指定保存文件的hdfs目录
a1.sinks.k1.hdfs.path = /flume/event/dirs
a1.sinks.k1.hdfs.filePrefix = hive-log
a1.sinks.k1.hdfs.rollSize = 10240
a1.sinks.k1.hdfs.rollInterval = 0
a1.sinks.k1.hdfs.rollCount = 0
#Specify the channel the sink should use
a1.sinks.k1.channel = c1

# Each channel's type is defined.
a1.channels.c1.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 1000
执行命令：bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/dir-mem-log.properties -Dflume.root.logger=INFO,console

【taildir 】既可以监控文件夹 也可以监控文件(按行读取文件夹文件中的内容可以实现递归读取，并提供数据恢复)

taildir.properties文件内容如下：

# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per a1, 
# in this case called 'a1'

a1.sources = s1
a1.channels = c1
a1.sinks = k1

# For each one of the sources, the type is defined
#目前版本为1.5编译的版本 1.7的版本才有taildir的类型 所以必须指定调用的具体类
a1.sources.s1.type = org.apache.flume.source.taildir.TaildirSource
#taildir_position.json该文件名不可改 该文件记录了文件读取的位置 避免下次启动进程的时候重新读取之前读取过的数据 进程启动的时候先加载这个json文件
a1.sources.s1.positionFile = /opt/datas/flume/taildir/position/taildir_position.json
a1.sources.s1.filegroups = f1 f2
#监控文件
a1.sources.s1.filegroups.f1 = /opt/datas/flume/taildir/hadoop14.txt
a1.sources.s1.headers.f1.headerKey1 = value1
#监控文件夹满足递归监控
a1.sources.s1.filegroups.f2 = /opt/datas/flume/taildir/hadoop/.*
a1.sources.s1.headers.f2.headerKey1 = value2
a1.sources.s1.headers.f2.headerKey2 = value2-2

# The channel can be defined as follows.
a1.sources.s1.channels = c1
a1.sinks.k1.channel = c1

# Each sink's type must be defined
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /flume/event/taildir
a1.sinks.k1.hdfs.filePrefix = hive-log
#Specify the channel the sink should use

# Each channel's type is defined.
a1.channels.c1.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 1000
执行命令：bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/taildir.properties -Dflume.root.logger=INFO,console


flume企业应用
-》扇出
在flume中有时候需要将一个源（source）将数据发送到多个地方（sink），在flume中该术语叫做扇出（fan out），也就是从一个source向多个channel，就是向多个sink传递事件，一个sink对应一个channel

sink.properties 文件内容如下：

# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per a1, 
# in this case called 'a1'

a1.sources = s1
a1.channels = c1 c2
a1.sinks = k1 k2

# For each one of the sources, the type is defined
a1.sources.s1.type = exec
a1.sources.s1.command=tail -f /opt/cdh5/hive-0.13.1-cdh5.3.6/logs/hive.log
# The channel can be defined as follows.
a1.sources.s1.channels = c1 c2
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2

# Each sink's type must be defined
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /flume/event/hdfs1
a1.sinks.k1.hdfs.filePrefix = hive-log1

a1.sinks.k2.type = hdfs
a1.sinks.k2.hdfs.path = /flume/event/hdfs2
a1.sinks.k2.hdfs.filePrefix = hive-log2
#Specify the channel the sink should use

# Each channel's type is defined.
a1.channels.c1.type = memory
a1.channels.c2.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 1000

a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 1000
执行命令：bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/sink.properties -Dflume.root.logger=INFO,console


-》扇入
在Flume中，可以通过配置第一组代理的sink为avro类型，并指向唯一的一个source为avro类型的代理来实现。这个代理将收到的数据合并，然后通过channel交由sink，到达最终的目的地。主要解决hdfs的高并发
先运行collect再运行client 防止数据丢失

avro-collect.properties文件内容如下：

# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per a1, 
# in this case called 'a1'
#汇总端的配置文件
a1.sources = s1
a1.channels = c1
a1.sinks = k1

# For each one of the sources, the type is defined
#指定汇总端的source类型 跟客户端的sink类型一致
a1.sources.s1.type = avro
a1.sources.s1.bind = 192.168.145.130
a1.sources.s1.port = 45454
# The channel can be defined as follows.
a1.sources.s1.channels = c1
a1.sinks.k1.channel = c1

# Each sink's type must be defined
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /flume/event/avro
a1.sinks.k1.hdfs.filePrefix = hive-log
a1.sinks.k1.hdfs.rollSize = 10240
a1.sinks.k1.hdfs.rollInterval = 0
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.useLocalTimeStamp = true
#Specify the channel the sink should use

# Each channel's type is defined.
a1.channels.c1.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 1000
执行命令：bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/avro-collect.properties -Dflume.root.logger=INFO,console


avro-client.properties文件内容如下：

# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per a1, 
# in this case called 'a1'
#客户端的配置文件
a1.sources = s1
a1.channels = c1
a1.sinks = k1

# For each one of the sources, the type is defined
a1.sources.s1.type = exec
a1.sources.s1.command=tail -f /opt/cdh5/hive-0.13.1-cdh5.3.6/logs/hive.log
# The channel can be defined as follows.
a1.sources.s1.channels = c1
a1.sinks.k1.channel = c1

# Each sink's type must be defined
#指定sink的类型
a1.sinks.k1.type = avro
#sink发到哪个机器的地址
a1.sinks.k1.hostname = 192.168.145.130
a1.sinks.k1.port = 45454
#Specify the channel the sink should use


# Each channel's type is defined.
a1.channels.c1.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
a1.channels.c1.capacity = 100
a1.channels.c1.transactionCapacity = 100
执行命令：bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/avro-client.properties -Dflume.root.logger=INFO,console


参考文献：http://blog.csdn.net/a2011480169/article/details/51544664








weixin_40652340
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
单节点搭建flume之exec和spooldir
Technology_2016的博客
04-12 5421
一、flume安装 1、上传flume包到master并解压缩 tar -zxvf flume-ng-1.2.0-cdh3u5.tar.gz 2、添加flume路径到.bash_profile中（可有可无，只为方便） export FLUME_HOME=/home/hadoop/flume-ng-1.2.0-cdh3u5 export FLUME_CONF_DIR=$FLUME_HOME
Flume企业开发案例
qq_43193797的博客
01-21 293
案例一：实时监控 Hive 日志，并上传到 HDFS 中   案例二：实时读取目录文件到 HDFS     案例三： 单数据源多出口案例(选择器) 案例四：单数据源多出口案例(Sink组) 一个channel连接多个sink，这些sink可以属于一个组，则这个组中的sink实现负载功能...
初探Flume—又一个分布式日志收集系统
12-26 6775
以前一直在使用Facebook开源的Scribe作为项目中的日志收集系统，Scribe能够满足我们需求，不过前段时间看到了几种分布式日志收集系统的比对之后，发现Cloudera的Flume从实现上来说更加完善，最近有时间探索了一下。 1 安装，作为Cloudera发行版的一大优势，安装从来不是什么麻烦的问题，只要我们将CDH的源（目前支持Ubuntu，CentOS，OpenSUSE等）加
Flume使用详解（三）
01-07
在Flume使用中配置相当重要，但也非常简单。 在conf目录下，创建一个配置文件，比如：template.conf（名字可以不固定,后缀也可以不固定） 相关配置： #配置Agent a1 的组件 a1.sources=r1 a1.channels=c1 (可以配置...
Flume使用详解（一）
01-07
概述： Flume最早是Cloudera提供的日志收集系统，后贡献给Apache。所以目前是Apache下的项目，...当前Flume有两个版本Flume 0.9X版本的统称Flume-og，老版本的flume需要引入zookeeper集群管理，性能 也较低（单线程工
Flume使用详解（四）
01-20
flume的Interceptors 概述： Flume有能力在运行阶段修改/删除Event，这是通过拦截器（Interceptors）来实现的。 拦截器需要实现org.apache.flume.interceptor.Interceptor接口。 拦截器可以修改或删除事件基于开发者...
Flume使用详解（二）
01-07
flume可以支持多级flume的agent，即flume可以前后相继形成多级的复杂流动，例如sink可以将数据写到下一个agent的source中，这样的话就可以连成串了，可以整体处理了。 置多个agent的数据流（多级流动） 数据流合并...
Flume部署和使用
01-07
Flume部署和使用 官方文档: http://flume.apache.org/ example: WebServer –> Agent[Source–>Channel–>Sink] –> HDFS 一.简介 Flume是一个分布式，可靠的的框架，它能从许多不同的数据源高效地收集、聚合和移动...
Hbase数据库的常用操作命令
热门推荐
weixin_40652340的博客
12-07 2万+
HBase的使用 1、自带了shell命令行 $ bin/hbase shell 2、输入help获取帮助信息 3、shell命令行的删除，需要按住ctrl+删除键才可以删除 4、list列出当前数据库中的表 5、list_namespace列出当前数据库中的所有namespace 相关Hbase数据库的常用操作命令：http://blog.csdn.net/scutshuxue
Hadoop之HA高可用性
weixin_40652340的博客
11-17 7637
HA存在的背景： HA的工作原理图： HDFS HA高可用性 1、active namenode对外提供服务和standby namenode时刻待机准备的 2、保证两个namenode任何时候都是元数据同步的 3、standby namenode同样需要去读取fsimage和edits文件 -》edits变化后的数据文件同样也是需要实时
Hive结合shell脚本实现自动化业务
weixin_40652340的博客
12-13 5551
【案例】hive脚本加载数据到hive分区表 access_logs/20170610/2017061000.log 2017061001.log 2017061002.log ......                      2017061023.log 二级分区：天/小时 crontab+shell 实现自动调度。 建库： create database loa
SparkSQL DSL开发
weixin_40652340的博客
01-30 2638
import org.apache.spark.sql.SQLContext import org.apache.spark.sql.expressions.Window import org.apache.spark.sql.hive.HiveContext import org.apache.spark.{SparkConf, SparkContext} case class Person
MapReduce之combine
weixin_40652340的博客
11-16 2353
Combine优化机制简介  MapReduce框架的运作基于键值对，即数据的输入是键值对，生成的结果也是存放在集合里的键值对，其中键值对的值也是一个集合，一个MapReduce任务的执行过程以及数据输入输出的类型如下所示，这里我们定义list表示集合：  　　map（K1， V1） -> list（K2， V2）  　　combine（K2， list（V2）） -> list（K2，
SparkCore
weixin_40652340的博客
01-29 1940
SparkCore ======================================== MapReduce   分布式计算框架   缺点：    -1.执行速度慢       IO瓶颈：磁盘IO、网络IO       shuffle的机制：数据需要输出到磁盘，而且每次都需要进行排序的操作    -2. 框架的缺陷       只有map和reduce两个操作
Hive日志分析案例
weixin_40652340的博客
12-04 1055
日志分析案例 1、分析流程 -》需求分析 -》数据采集 -》数据清洗 -》数据分析 -》结果展示 2、【需求分析】 -》日期 -》按照日期进行分组，基于时间维度的分析 -》可以考虑建分区表，分区给定日期 -》登录人数，代表的就是会员，使用账号登录的人 -》登录之后会产生一个会员ID -》处理的时候可以看会员ID有没有值 -》PV -》count(url) -》UV
Kafka
weixin_40652340的博客
02-01 1045
====================================================================== 根据官网的介绍，ApacheKafka®是一个分布式流媒体平台，它主要有3种功能： 　　1：发布和订阅消息流，这个功能类似于消息队列，这也是kafka归类为消息队列框架的原因 　　2：以容错的方式记录消息流，kafka以文件的方式来存储消息
HUE简介及部署集成
weixin_40652340的博客
12-26 992
一：Hue简介及核心功能描述 Hue是一个开源的Apache Hadoop UI系统，由Cloudera Desktop演化而来，最后Cloudera公司将其贡献给Apache基金会的Hadoop社区，它是基于Python Web框架Django实现的。 通过使用Hue我们可以在浏览器端的Web控制台上与Hadoop集群进行交互来分析处理数据，例如操作HDFS上的数据，运行MapRed
Oozie的使用
weixin_40652340的博客
12-25 830
一：Oozie运行examples。 1、解压示例包 $ tar -zxf oozie-examples.tar.gz -C ./ 2、$ bin/hdfs dfs -put /opt/cdh5/oozie-4.0.0-cdh5.3.6/examples/ 3、编写job.properties # http://www.apache.org/licenses/LICENSE-2
Hadoop之历史服务器与日志聚合和文件权限
weixin_40652340的博客
11-15 604
1、历史服务器是一个轻量级的 2、 配置mapred-site.xml文件，指定两个参数 mapreduce.jobhistory.address bigdata-01.yushu.com:10020 mapreduce.jobhistory.webapp.address bigdata-01.yushu.com:19888 3、启动： $ sbin/mr-job
java使用flume自定义Handler
最新发布
01-31
在Java中使用Flume自定义Handler，你需要按照以下步骤进行操作： 1. 首先，在你的Java项目中引入Flume的依赖。在你的pom.xml文件中添加以下代码： ```xml <groupId>org.apache.flume <artifactId>flume-ng-core ...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值