zabbix+flume+hadoop安装及其监控

最新推荐文章于 2023-10-05 20:25:51 发布

poisions

最新推荐文章于 2023-10-05 20:25:51 发布

阅读量1.3k

点赞数

分类专栏： Flume hadoop学习

Flume 同时被 2 个专栏收录

3 篇文章 0 订阅

订阅专栏

hadoop学习

3 篇文章 0 订阅

订阅专栏

1 Hadoop 2.7.1安装部署

1.1 相关链接地址

安装集群步骤：

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html

配置文件默认说明地址：

http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml

http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

http://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-common/yarn-default.xml

webhdfs使用说明文档：

http://hadoop.apache.org/docs/r1.0.4/webhdfs.html

或下载Hadoop包解压后根目录 /doc/hadoop-yarn/hadoop-yarn-site/WebHDFS.html

1.2 创建用户

[root@iZ94bxra9qaZ ~]# useradd yq

1.3 用户赋权限

[root@iZ94bxra9qaZ ~]# chown yq /home/yq/

[root@iZ94bxra9qaZ ~]# chmod -R 777 /home/yq/

1.4 配置为可以无密码登录本机

如果没有ssh 则需要安装机器可以联网使用 sudo apt-getinstall ssh 命令即可以安装。

查看yq用户下是否存在.ssh文件夹

[root@iZ94bxra9qaZ ~]# ls -a /home/yq/

. .. .bash_logout .bash_profile .bashrc

如果没有，手动创建一个

[yq@iZ94bxra9qaZ ~]$ ssh-keygen -t rsa -f~/.ssh/id_rsa

或者

[yq@iZ94bxra9qaZ ~]$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

Generating public/private dsa key pair.

Your identification has been saved in/home/yq/.ssh/id_dsa.

Your public key has been saved in/home/yq/.ssh/id_dsa.pub.

The key fingerprint is:

71:c1:60:84:36:65:7b:8c:c5:89:20:49:11:3d:73:a5yq@iZ94bxra9qaZ

The key's randomart image is:

+--[ DSA 1024]----+

| .==.=**+. |

| ..Boo*+. |

| . =E + |

| + |

| S |

| |

+-----------------+

[yq@iZ94bxra9qaZ ~]$

ssh-keygen 代表生成密钥：-t 表示指定生产的密钥类型；dsa是dsa密钥认证的意思，即密钥类型；-P用于提供蜜语；-f指定生成的密钥文件。

把id_dsa.pub (公钥)追加到授权的key里面去

命令为：

[yq@iZ94bxra9qaZ ~]$ cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys

或者

[yq@iZ94bxra9qaZ ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

另外 /etc/ssh/sshd_config 中下列三行去除注释

RSAAuthentication yes

PubkeyAuthentication yes

AuthorizedKeysFile .ssh/authorized_keys

然后执行service sshd restart

[root@localhost ssh]# service sshd restart

1.5 配置hosts

配置/etc/hosts

master为ye.hadoop

slave为ye.hadoop1

我这里两台机器

127.0.0.1 localhost localhost.localdomainlocalhost4 localhost4.localdomain4

::1 localhostlocalhost.localdomain localhost6 localhost6.localdomain6

192.168.60.128 ye.hadoop

192.168.60.129 ye.hadoop1

配置让master可以访问slave

把ye.hadoop的公钥拿到ye.hadoop1机器上

[yq@localhost .ssh]$ scp yq@ye.hadoop:~/.ssh/id_rsa.pub./master_rsa.pub

id_rsa.pub 100% 406 0.4KB/s 00:00

[yq@localhost .ssh]$ ls

authorized_keys id_rsa id_rsa.pub known_hosts master_rsa.pub

[yq@localhost .ssh]$ cat master_rsa.pub >>authorized_keys

1.6 测试是否能免密码登录

[yq@localhost .ssh]$ ssh ye.hadoop

Last login: Sat Jan 30 09:39:34 2016 from ye.hadoop1

[yq@ye ~]$

1.7 安装hadoop

[yq@ye ~]$ tar -xzvf hadoop-2.7.1.tar.gz

root 用户下：

[root@localhost yq]# rpm -ivh jdk-7u21-linux-x64.rpm

[root@localhost usr]# cd /etc/

[root@localhost usr]vi /etc/profile

export JAVA_HOME=/usr/Java/jdk1.7.0_21

export JRE_HOME=$JAVA_HOME/jre

export CLASSPATH=$JAVA_HOME/lib

export PATH=$PATH:/usr/java/jdk1.7.0_21/bin

export HADOOP_HOME=/home/yq/hadoop-2.7.1

exportPATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export HADOOP_PREFIX=/home/yq/hadoop-2.7.1

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export HADOOP_HDFS_HOME=$HADOOP_HOME

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_YARN_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

exportHADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

exportCLASSPATH=$CLASSPATH:$HADOOP_HOME/libexec/share/hadoop/common/hadoop-common-2.7.1.jar

exportCLASSPATH=$CLASSPATH:$HADOOP_HOME/libexec/share/hadoop/common/hadoop-nfs-2.7.1.jar

[root@localhost etc]# source /etc/profile

配置core-site.xml、hdfs-site.xml、yarn-site.xml、mapred-site.xml 文件，参考官方网址配置。

检测查看hadoop 环境变量

[yq@ye hadoop-2.7.1]$ bin/hadoop classpath

/home/yq/hadoop-2.7.1/libexec/etc/hadoop:/home/yq/hadoop-2.7.1/share/hadoop/common/lib/*:/home/yq/hadoop-2.7.1/share/hadoop/common/*:/home/yq/hadoop-2.7.1/share/hadoop/hdfs:/home/yq/hadoop-2.7.1/share/hadoop/hdfs/lib/*:/home/yq/hadoop-2.7.1/share/hadoop/hdfs/*:/home/yq/hadoop-2.7.1/share/hadoop/yarn/lib/*:/home/yq/hadoop-2.7.1/share/hadoop/yarn/*:/home/yq/hadoop-2.7.1/share/hadoop/mapreduce/lib/*:/home/yq/hadoop-2.7.1/share/hadoop/mapreduce/*

[yq@ye hadoop-2.7.1]$

向节点 ye.hadoop1复制配置好的hadoop

scp -r hadoop-2.7.1 yq@ye.hadoop1:/home/yq/hadoop-2.7.1

在ye.hadoop上格式化namenode

[yq@ye hadoop-2.7.1]$ bin/hdfs namenode -format

启动hadoop

[yq@ye hadoop-2.7.1]$ sbin/start-all.sh

访问地址

http://192.168.60.128:50070

停止hadoop

[yq@ye hadoop-2.7.1]$ sbin/stop-all.sh

1.8 遇到问题

非root用户可能会遇到权限问题

当你去认证登录时候还是要输入密码认证

当我去查看安全日志时有这样一行错误

cat /var/log/secure

Jan 30 08:01:34 localhost sshd[33404]:Authentication refused: bad ownership or modes for directory /home/yq

Jan 30 08:04:35 localhost sshd[33427]:Authentication refused: bad ownership or modes for directory /home/yq

解决此问题方法

在用户yq下给/home/yq 赋700的权限

chmod 700 /home/yq

当你再去ssh -vvv localhost时可能会报~/.ssh/下某一个文件 644权限太高的错误解决此办法

chmod 600 ~/.ssh/*

2 Flume 1.6安装部署

安装了hadoop之后，安装flume非常简单。

2.1 下载安装包

去官网下载apache-flume-1.6.0-bin.tar.gz

解压

[yq@ye ~]$ tar zxf apache-flume-1.6.0-bin.tar.gz

2.2 修改flume-env.sh和flume-http.conf文件

[yq@ye ~]$ cd apache-flume-1.6.0-bin

[yq@ye apache-flume-1.6.0-bin]$ ls

bin CHANGELOG conf DEVNOTES docs lib LICENSE NOTICE README RELEASE-NOTES tools

[yq@ye apache-flume-1.6.0-bin]$ cd conf/

[yq@ye conf]$ pwd

/home/yq/apache-flume-1.6.0-bin/conf

[yq@ye conf]$ mv flume-env.sh.template flume-env.sh

[yq@ye conf]$ Vi flume-env.sh

修改标红处

# Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements. See the NOTICE file

# distributed with this work for additional information

# regarding copyright ownership. The ASF licenses this file

# to you under the Apache License, Version 2.0 (the

# "License"); you may not use this file except incompliance

# with the License. You mayobtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing,software

# distributed under the License is distributed on an "ASIS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express orimplied.

# See the License for the specific language governing permissionsand

# limitations under the License.

# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will besourced

# during Flume startup.

# Enviroment variables can be set here.

export JAVA_HOME=/usr/java/jdk1.7.0_21

# Give Flume more memory and pre-allocate, enable remote monitoringvia JMX

export JAVA_OPTS="-Xms100m -Xmx2000m-Dcom.sun.management.jmxremote"

# Note that the Flume conf directory is always included in theclasspath.

#FLUME_CLASSPATH=""

[yq@ye conf]$ mv flume-conf.properties.template flume-http.conf

[yq@ye conf]$ vi flume-http.conf

# KIND, either express or implied. See the License for the

# specific language governing permissions and limitations

# under the License.

# The configuration file needs to define the sources,

# the channels and the sinks.

# Sources, channels and sinks are defined per agent,

# in this case called 'agent'

a1.sources = r1

a1.channels = c1

a1.sinks = s1

a1.sources.r1.type = http

a1.sources.r1.bind = 10.10.2.236

a1.sources.r1.port = 5140

a1.sources.r1.handler = org.apache.flume.source.http.JSONHandler

a1.sources.r1.channels = c1

a1.sources.r1.interceptors=i2

a1.sources.r1.interceptors.i2.type=timestamp

a1.channels.c1.type=memory

a1.channels.c1.capacity=10000

a1.channels.c1.transactionCapacity=1000

a1.channels.c1.keep-alive=30

a1.sinks.s1.type=hdfs

a1.sinks.s1.channel=c1

a1.sinks.s1.hdfs.path=hdfs://ye.hadoop:9000/flume/events/%y-%m-%d

a1.sinks.s1.hdfs.fileType=DataStream

a1.sinks.s1.hdfs.writeFormat=Text

a1.sinks.s1.hdfs.timeZone=Asia/Shanghai

a1.sinks.s1.hdfs.rollInterval=0

a1.sinks.s1.hdfs.rollSize=65011712

a1.sinks.s1.hdfs.rollCount=0

a1.sinks.s1.hdfs.idleTimeout=60

2.3 启动flume

[yq@ye apache-flume-1.6.0-bin]$ pwd

/home/yq/apache-flume-1.6.0-bin

[yq@ye apache-flume-1.6.0-bin]$ bin/flume-ng agent --conf conf --conf-fileconf/flume-http.conf --name a1 -Dflume.monitoring.type=http-Dflume.monitoring.port=5240 -Dflume.root.logger=INFO,console

使用配置文件为conf/flume-http.conf

-Dflume.monitoring.type=http-Dflume.monitoring.port=5240 为计数器端口，提供给之后zabbix监控使用。

3 Zabbix 3.0安装部署

Zabbix 3.0最低需要PHP5.4版本

3.1 安装yum

3.1.1 卸载原有yum

[root@localhost ~]# rpm -aq | grep yum |xargs rpm -e --nodeps

3.1.2 所需包下载地址：

https://lug.ustc.edu.cn/wiki/_export/code/mirrors/help/centos?codeblock=2%20-O%20/etc/yum.repos.d/CentOS-Base.repo

http://centos.ustc.edu.cn/centos/6/os/x86_64/Packages/python-iniparse-0.3.1-2.1.el6.noarch.rpm

http://centos.ustc.edu.cn/centos/6/os/x86_64/Packages/yum-plugin-fastestmirror-1.1.30-30.el6.noarch.rpm

http://centos.ustc.edu.cn/centos/6/os/x86_64/Packages/yum-metadata-parser-1.1.2-16.el6.x86_64.rpm

http://thunder.mirrors.ustc.edu.cn/centos/6/os/x86_64/Packages/yum-3.2.29-69.el6.centos.noarch.rpm

3.1.3 安装rpm包

[root@localhost home]# rpm -ivhpython-iniparse-0.3.1-2.1.el6.noarch.rpm

Preparing... ########################################### [100%]

packagepython-iniparse-0.3.1-2.1.el6.noarch is already installed

[root@localhost home]# rpm -ivh yum-*

Preparing... ########################################### [100%]

1:yum-metadata-parser ################################## ( 80%)

###########################################[ 33%]

2:yum-plugin-fastestmirro########################################### [67%]

3:yum ########################################### [100%]

[root@localhost home]#

[root@localhost home]# rpm -Uvhhttp://mirror.webtatic.com/yum/el6/latest.rpm

Retrievinghttp://mirror.webtatic.com/yum/el6/latest.rpm

warning: /var/tmp/rpm-tmp.snnf1M: Header V4DSA/SHA1 Signature, key ID cf4c4ff9: NOKEY

Preparing... ###########################################[100%]

1:webtatic-release ########################################### [100%]

[root@localhost home]# rpm -ivhhttp://mirrors.ustc.edu.cn/epel/6/x86_64/epel-release-6-8.noarch.rpm

Retrievinghttp://mirrors.ustc.edu.cn/epel/6/x86_64/epel-release-6-8.noarch.rpm

warning: /var/tmp/rpm-tmp.Xyd3s5: Header V3RSA/SHA256 Signature, key ID 0608b895: NOKEY

Preparing... ########################################### [100%]

packageepel-release-6-8.noarch is already installed

[root@localhost home]# rpm -ivhius-release-1.0-14.ius.el6.noarch.rpm

warning:ius-release-1.0-14.ius.el6.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID9cd4953f: NOKEY

Preparing... ########################################### [100%]

packageius-release-1.0-14.ius.el6.noarch is already installed

3.1.4 修改CentOS-Base.repo

[root@localhost home]# cp CentOS-Base.repo/etc/yum.repos.d/CentOS-Base.repo

cp: overwrite`/etc/yum.repos.d/CentOS-Base.repo'? yes

[root@localhost home]#

[root@localhost home]# sed -i's/\$releasever/6/g' /etc/yum.repos.d/CentOS-Base.repo

[root@localhost home]#

[root@localhost home]# yum makecache

3.1.5 遇到问题

Error:Cannot retrieve metalink for repository: epel. Please verify its path and tryagain

解决办法

[root@localhost home]# su-c "sed -i 's|^#baseurl|baseurl| ; s|^mirrorlist|#mirrorlist|'/etc/yum.repos.d/*"

机器不能访问https造成

是因为DNS配置有问题

修改/etc/resolv.conf文件

3.2 安装lamp环境

[root@ye ~]#yuminstall list |grep PHP

[root@ye ~]# yum -y install php54w-MySQL.x86_64 php54w-gd.x86_64 php54w-mbstring.x86_64 php54w-odbc.x86_64php54w-xml.x86_64 php54w-pear.noarch

[root@ye ~]# yum -y install php54w-bcmath.x86_64

3.3 安装zabbix所需组件

[root@ye ~]# yum install -y curl curl-devel net-snmpnet-snmp-devel perl-DBI

3.4 创建用户

[root@localhost Desktop]# useradd zabbix

3.5 文件夹赋权

[root@ye mysql]# chown zabbix /home/zabbix

[root@ye mysql]#

[root@ye mysql]# chmod -R 777 /home/zabbix

3.6 解压zabbix

zabbix 下载地址：

http://iweb.dl.sourceforge.net/project/zabbix/ZABBIX%20Latest%20Development/3.0.0alpha2/zabbix-3.0.0alpha2.tar.gz

[root@ye zabbix]# ls

zabbix-3.0.0alpha2 (1).tar.gz

[root@ye zabbix]# tar xfzabbix-3.0.0alpha2\ 1 .tar.gz

[root@localhost~]#mysql

mysql>create database zabbix;

mysql>grant all on zabbix.* tozabbix@localhost identified by '123456';

mysql>use zabbix;

mysql>source /home/zabbix/zabbix-3.0.0alpha2/database/mysql/schema.sql

mysql>source/home/zabbix/zabbix-3.0.0alpha2/database/mysql/images.sql

mysql> source/home/zabbix/zabbix-3.0.0alpha2/database/mysql/data.sql

mysql>exit

3.7 编译安装

[root@ye zabbix-3.0.0alpha2]# pwd

/home/zabbix/zabbix-3.0.0alpha2

[root@ye zabbix-3.0.0alpha2]# ./configure--enable-server --enable-agent --with-mysql --with-net-snmp --with-libcurl

参数说明
—prefix=/usr/local/zabbix为指定安装目录为/usr/local/zabbix
—enable-server为安装zabbix服务端程序
—enable-agent为安装agent程序
—with-mysql为使用mysql数据库
—enable-ipv6为启用ipv6支持
—with-net-snmp为启用snmp支持
—with-libcurl为启用curl