一、简介
官网:Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data . It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.
flume是一个分布式、可靠和可用的服务,有效地收集、聚合和移动大量的日志数据,它有一个简单而灵活的架构基于流媒体数据流,这是健壮的可靠性和容错可调机制和许多故障转移和恢复机制,它使用一个简单的可扩展的数据模型,允许在线分析应用程序。
简单处理示意图
二、下载安装
官方下载地址:http://flume.apache.org/download.html
wget "http://mirrors.cnnic.cn/apache/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz"
tar -xzvf apache-flume-1.6.0-bin.tar.gz
下载后直接解压。。。就算安装完毕
三、配置
环境配置: cat /etc/profile
flume启动文件配置:进入flume的安装目录 cd /data1/flume/apache-flume-1.6.0-bin/conf/
新建文件:example.conf
文件内容:
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
启动命名:
bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console