大数据学习5之分布式日志收集框架Flume——Flume环境部署

Flume环境部署

Flume安装前置条件

Flume官网-> Documation->user guide->introduction->system requirements

1. Java运行时环境-Java 1.8或更高版本
2. 内存-足够的内存，用于源，通道或接收器使用的配置
3. 磁盘空间-足够的磁盘空间用于通道或接收器使用的配置
4. 目录权限-代理使用的目录的读/写权限

• Java Runtime Environment - Java 1.8 or later Memory - Sufficient
• memory for configurations used by sources, channels or sinks Disk
• Space - Sufficient disk space for configurations used by channels
  or sinks
• Directory Permissions - Read/Write permissions for
  directories used by agent

1.安装JDK

（1）进入software目录找到jdk压缩包，解压到app目录

tar -zxvf jdk-8u144-linux-x64.tar.gz -C ~/app/

[hadoop@hadoop000 Desktop]$ pwd
/home/hadoop/Desktop
[hadoop@hadoop000 Desktop]$ cd ~
[hadoop@hadoop000 ~]$ pwd
/home/hadoop
[hadoop@hadoop000 ~]$ ls
app      Documents    lib          Public    source     Videos
data     Downloads    maven_repos  shell     Templates
Desktop  install.log  Pictures     software  tmp
[hadoop@hadoop000 ~]$ cd software
[hadoop@hadoop000 software]$ ls
apache-maven-3.3.9-bin.tar.gz   kafka_2.11-0.9.0.0.tgz
flume-ng-1.6.0-cdh5.7.0.tar.gz  mysql-connector-java-5.1.27-bin.jar
hadoop-2.6.0-cdh5.7.0.tar.gz    scala-2.11.8.tgz
hbase-1.2.0-cdh5.7.0.tar.gz     spark-2.2.0-bin-2.6.0-cdh5.7.0.tgz
jdk-8u144-linux-x64.tar.gz      zookeeper-3.4.5-cdh5.7.0.tar.gz
[hadoop@hadoop000 software]$ tar -zxvf jdk-8u144-linux-x64.tar.gz -C ~/app/

（2）进入app目录的jdk1.8.0_144解压后的jdk包，配置到系统环境变量~/.bash_profile中,改动过后source一下使其生效source ~/.bash_profile

vi ~/.bash_profile

export JAVA_HOME=/home/hadoop/app/jdk1.8.0_144
export PATH=$JAVA_HOME/bin:$PATH

source ~/.bash_profile

[hadoop@hadoop000 software]$ cd ~/app
[hadoop@hadoop000 app]$ ls
apache-flume-1.6.0-cdh5.7.0-bin  kafka_2.11-0.9.0.0
apache-maven-3.3.9               scala-2.11.8
data                             spark-2.2.0-bin-2.6.0-cdh5.7.0
hadoop-2.6.0-cdh5.7.0            tmp
hbase-1.2.0-cdh5.7.0             zookeeper-3.4.5-cdh5.7.0
jdk1.8.0_144
[hadoop@hadoop000 app]$ cd jdk1.8.0_144/
[hadoop@hadoop000 jdk1.8.0_144]$ ls
bin        javafx-src.zip  man          THIRDPARTYLICENSEREADME-JAVAFX.txt
COPYRIGHT  jre             README.html  THIRDPARTYLICENSEREADME.txt
db         lib             release
include    LICENSE         src.zip
[hadoop@hadoop000 jdk1.8.0_144]$ pwd
/home/hadoop/app/jdk1.8.0_144
[hadoop@hadoop000 jdk1.8.0_144]$ vi ~/.bash_profile
[hadoop@hadoop000 jdk1.8.0_144]$ source ~/.bash_profile

验证一下

[hadoop@hadoop000 jdk1.8.0_144]$ java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

2.安装Flume

（1）下载

（2）解压到app目录下

tar -zxvf flume-ng-1.6.0-cdh5.7.0.tar.gz -C ~/app/

（3）将app目录下解压后的flume软件目录配置到系统环境变量中

vi ~/.bash_profile
在这里插入图片描述

export FLUME_HOME=/home/hadoop/app/apache-flume-1.6.0-cdh5.7.0-bin
export PATH=$FLUME_HOME/bin:$PATH

(4)source一下系统环境变量文件，使其生效

source ~/.bash_profile

验证一下

[hadoop@hadoop000 app]$ cd $FLUME_HOME
[hadoop@hadoop000 apache-flume-1.6.0-cdh5.7.0-bin]$ 

（5）配置flume

进入flume软件的conf文件夹下，拷贝flume-env.sh.template文件，文件名为flume-env.sh，导出JAVA_HOME.
JAVA_HOME可以通过echo $JAVA_HOME进行输出。

export JAVA_HOME=/home/hadoop/app/jdk1.8.0_144

验证一下：

flume-ng version

此时，使用flume-ng命令，flume-ng [options]…
<> 代表必填的，[]代表是选填的

[hadoop@hadoop000 ~]$ flume-ng
Error: Unknown or unspecified command ''

Usage: /home/hadoop/app/apache-flume-1.6.0-cdh5.7.0-bin/bin/flume-ng <command> [options]...

commands:
  help                      display this help text
  agent                     run a Flume agent
  avro-client               run an avro Flume client
  version                   show Flume version info

global options:
  --conf,-c <conf>          use configs in <conf> directory
  --classpath,-C <cp>       append to the classpath
  --dryrun,-d               do not actually start Flume, just print the command
  --plugins-path <dirs>     colon-separated list of plugins.d directories. See the
                            plugins.d section in the user guide for more details.
                            Default: $FLUME_HOME/plugins.d
  -Dproperty=value          sets a Java system property value
  -Xproperty=value          sets a Java -X option

agent options:
  --name,-n <name>          the name of this agent (required)
  --conf-file,-f <file>     specify a config file (required if -z missing)
  --zkConnString,-z <str>   specify the ZooKeeper connection to use (required if -f missing)
  --zkBasePath,-p <path>    specify the base path in ZooKeeper for agent configs
  --no-reload-conf          do not reload config file if changed
  --help,-h                 display help text

avro-client options:
  --rpcProps,-P <file>   RPC client properties file with server connection params
  --host,-H <host>       hostname to which events will be sent
  --port,-p <port>       port of the avro source
  --dirname <dir>        directory to stream to avro source
  --filename,-F <file>   text file to stream to avro source (default: std input)
  --headerFile,-R <file> File containing event headers as key/value pairs on each new line
  --help,-h              display help text

  Either --rpcProps or both --host and --port must be specified.

Note that if <conf> directory is specified, then it is always included first
in the classpath.