flume单机版环境搭建

flume逻辑上采用三层架构：

Agent层：用于采集数据，可用于Flume存储数据流，同时Agent将产生的数据传输到Collector；

Collector层：其作用是汇总多个Agent上报的数据并加载到Storage中，在多个Collector之间遵循负载均衡规则，在集群环境下该层体现比较明显，单机版展现不出来；

Storage层：是一个存储系统，类型不固定，可以是文本文件、HDFS、Hbase等。

 

flume的核心由Source、Channel、Sink三个组件构成：

Source:收集数据源，并传输给Channel.

Channel:用户中转临时存储，保存所有Source组件上报的数据。

Sink:从Channel中读取数据（若读取成功，会删除Channel中的缓存数据），然后发送给存储介质。

单机环境搭建

下载安装包：官网1.8.0版本

cd /data/soft/new/flume
//解压
tar -zxvf apache-flume-1.8.0-bin.tar.gz

cd apache-flume-1.8.0-bin/conf
//修改flume的jdk环境变量
cp flume-env.sh.template flume-env.sh
vim flume-env.sh
#增加JAVA_HOME
export JAVA_HOME=/data/soft/new/jdk/jdk1.8.0_11
:wq
cp flume-conf.properties.template flume-conf.properties
vim flume-conf.properties

flume-conf.properties 内容如下

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#  http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.


# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per agent, 
# in this case called 'agent'

agent.sources = source1
agent.channels = channel1
agent.sinks = sink1

# For each one of the sources, the type is defined
agent.sources.source1.type = spooldir

# The channel can be defined as follows.
agent.sources.source1.channels = channel1
agent.sources.source1.spoolDir = /data/flume/logdfs
agent.sources.source1.fileHeader = false
agent.sources.source1.interceptors = i1
agent.sources.source1.interceptors.i1.type = timestamp

# Each sink's type must be defined
agent.sinks.loggerSink.type = logger

#Specify the channel the sink should use
agent.sinks.loggerSink.channel = channel1

# Each channel's type is defined.
agent.channels.channel1.type = file
agent.channels.channel1.checkpointDir = /data/flume/logdfs
agent.channels.channel1.dataDir = /data/flume/logdfstmp

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
#agent.channels.channel1.capacity = 100
agent.sinks.sink1.type = hdfs
agent.sinks.sink1.hdfs.path = /data/flume/logdfs
agent.sinks.sink1.hdfs.fileType = DataStream
agent.sinks.sink1.hdfs.writeFormat = Text
agent.sinks.sink1.hdfs.rollInterval = 1
agent.sinks.sink1.channel = channel1
agent.sinks.sink1.hdfs.filePrefix = %Y-%m-%d

设置环境变量

vim /etc/profile
export FLUME_HOME=/data/soft/new/flume/apache-flume-1.8.0-bin
export PATH=$PATH:$JAVA_HOME/bin:$FLUME_HOME/bin
:wq
source /etc/profile

执行命令启动

flume-ng agent -n agent -c conf -f /data/soft/new/flume/apache-flume-1.8.0-bin/conf/flume-conf.properties

flume就启动成功了，会将/data/flume/logdfs目录下文件收集起来，重命名为****.COMPLETED并上传到hdfs

参考：Flume数据采集之常见集群配置案例