Flume原理、安装和使用
1.flume是分布式的日志收集系统,把收集来的数据传送到目的地去。
2.flume里面有个核心概念,叫做agent。agent是一个java进程,运行在日志收集节点。
3.agent里面包含3个核心组件:source、channel、sink。
3.1 source组件是专用于收集日志的,可以处理各种类型各种格式的日志数据,包括avro、thrift、exec、jms、spooling directory、netcat、sequence generator、syslog、http、legacy、自定义。
source组件把数据收集来以后,临时存放在channel中。
3.2 channel组件是在agent中专用于临时存储数据的,可以存放在memory、jdbc、file、自定义。
channel中的数据只有在sink发送成功之后才会被删除。
3.3 sink组件是用于把数据发送到目的地的组件,目的地包括hdfs、logger、avro、thrift、ipc、file、null、hbase、solr、自定义。
4.在整个数据传输过程中,流动的是event。事务保证是在event级别。
5.flume可以支持多级flume的agent,支持扇入(fan-in)、扇出(fan-out)。
安装
1、先解压
[root@slave2 ~]# tar -xvf apache-flume-1.7.0-bin.tar -C /opt/
2、修改配置文件
[root@slave2 apache-flume-1.7.0-bin]# cd conf/
[root@slave2 conf]# ls
flume-conf.properties.template flume-env.ps1.template flume-env.sh.template log4j.properties
[root@slave2 conf]# ll
total 16
-rw-r--r--. 1 root root 1661 Sep 26 06:49 flume-conf.properties.template
-rw-r--r--. 1 root root 1455 Sep 26 06:49 flume-env.ps1.template
-rw-r--r--. 1 root root 1565 Sep 26 06:49 flume-env.sh.template
-rw-r--r--. 1 root root 3107 Sep 26 06:49 log4j.properties
[root@slave2 conf]# cp flume-env.sh.template flume-env.sh
[root@slave2 conf]# cp flume-conf.properties.template flume-conf.properties
3、修改系统配置文件
在/etc/profile增加
export FLUME_HOME=/opt/apache-flume-1.7.0-bin
export FLUME_CONF=$FLUME_HOME/conf
export PATH=$PATH:$FLUME_HOME/bin
4、修改JAVA_HOME和JAVA_OPTS
[root@slave2 conf]# vim flume-env.sh
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced
# during Flume startup.
# Enviroment variables can be set here.
export JAVA