Flume安装及测试

一、简介

       官网：Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data . It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application. 

     flume是一个分布式、可靠和可用的服务,有效地收集、聚合和移动大量的日志数据，它有一个简单而灵活的架构基于流媒体数据流，这是健壮的可靠性和容错可调机制和许多故障转移和恢复机制，它使用一个简单的可扩展的数据模型,允许在线分析应用程序。

     简单处理示意图

     

二、下载安装

官方下载地址：http://flume.apache.org/download.html

  wget "http://mirrors.cnnic.cn/apache/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz"
  tar -xzvf apache-flume-1.6.0-bin.tar.gz

   下载后直接解压。。。就算安装完毕

三、配置

       环境配置：  cat /etc/profile

       

      flume启动文件配置：进入flume的安装目录  cd /data1/flume/apache-flume-1.6.0-bin/conf/

      新建文件：example.conf

      文件内容：

  # Name the components on this agent
  a1.sources = r1
  a1.sinks = k1
  a1.channels = c1

  # Describe/configure the source
  a1.sources.r1.type = netcat
  a1.sources.r1.bind = localhost
  a1.sources.r1.port = 44444

  # Describe the sink
  a1.sinks.k1.type = logger

  # Use a channel which buffers events in memory
  a1.channels.c1.type = memory
  a1.channels.c1.capacity = 1000
  a1.channels.c1.transactionCapacity = 100

  # Bind the source and sink to the channel
  a1.sources.r1.channels = c1
  a1.sinks.k1.channel = c1

启动命名：

bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console