Flume的安装和测试

Flume原理、安装和使用
1.flume是分布式的日志收集系统，把收集来的数据传送到目的地去。
2.flume里面有个核心概念，叫做agent。agent是一个java进程，运行在日志收集节点。
3.agent里面包含3个核心组件：source、channel、sink。
3.1 source组件是专用于收集日志的，可以处理各种类型各种格式的日志数据,包括avro、thrift、exec、jms、spooling directory、netcat、sequence generator、syslog、http、legacy、自定义。
source组件把数据收集来以后，临时存放在channel中。
3.2 channel组件是在agent中专用于临时存储数据的，可以存放在memory、jdbc、file、自定义。
channel中的数据只有在sink发送成功之后才会被删除。
3.3 sink组件是用于把数据发送到目的地的组件，目的地包括hdfs、logger、avro、thrift、ipc、file、null、hbase、solr、自定义。
4.在整个数据传输过程中，流动的是event。事务保证是在event级别。
5.flume可以支持多级flume的agent，支持扇入(fan-in)、扇出(fan-out)。

安装
1、先解压

[root@slave2 ~]# tar -xvf apache-flume-1.7.0-bin.tar -C /opt/

2、修改配置文件

[root@slave2 apache-flume-1.7.0-bin]# cd conf/
[root@slave2 conf]# ls
flume-conf.properties.template  flume-env.ps1.template  flume-env.sh.template  log4j.properties
[root@slave2 conf]# ll
total 16
-rw-r--r--. 1 root root 1661 Sep 26 06:49 flume-conf.properties.template
-rw-r--r--. 1 root root 1455 Sep 26 06:49 flume-env.ps1.template
-rw-r--r--. 1 root root 1565 Sep 26 06:49 flume-env.sh.template
-rw-r--r--. 1 root root 3107 Sep 26 06:49 log4j.properties
[root@slave2 conf]# cp flume-env.sh.template flume-env.sh
[root@slave2 conf]# cp flume-conf.properties.template  flume-conf.properties

3、修改系统配置文件
在/etc/profile增加

export FLUME_HOME=/opt/apache-flume-1.7.0-bin
export FLUME_CONF=$FLUME_HOME/conf
export PATH=$PATH:$FLUME_HOME/bin

4、修改JAVA_HOME和JAVA_OPTS

[root@slave2 conf]# vim flume-env.sh

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced
# during Flume startup.

# Enviroment variables can be set here.

 export JAVA