tar包安装cdh

CDH

CDH安装方式

Cloudera Manager

Yum

Rpm

Tarball

Cloudera Manager

Cloudera Manager是一个管理CDH的端到端的应用

作用：

管理

监控

诊断

集成

架构

Server
    管理控制台服务器和应用程序逻辑
    负责软件安装、 配置
    启动和停止服务
    管理服务运行的群集

Agent
    安装在每台主机上
    负责启动和停止进程，配置，监控主机

Management Service
    由一组角色组成的服务，执行各种监视、报警和报告功能

Database
    数据库支撑

Cloudera Repository
    组件、软件下载的文档

Clients

Admin Console

API

Cloudera Manager安装前系统环境准备

1、网络配置

vi /etc/sysconfig/network
vi /etc/hosts
192.168.230.100 cdh1
192.168.230.101 cdh2
192.168.230.102 cdh3

vi /etc/sysconfig/network-scripts/ifcfg-eth0
vi /etc/resolv.conf

2、SSH免密钥登录

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

cdh1发送公钥给cdh2、cdh3
[root@cdh1 ~]# scp ~/.ssh/id_dsa.pub root@cdh2:/opt/
[root@cdh1 ~]# scp ~/.ssh/id_dsa.pub root@cdh3:/opt/

cdh2、cdh3将cdh1发过来的公钥加入到认证文件中去
[root@cdh2 ~]$ cat /opt/id_dsa.pub >> ~/.ssh/authorized_keys
[root@cdh3 ~]$ cat /opt/id_dsa.pub >> ~/.ssh/authorized_keys

3、防火墙关闭

查看防火墙状态
service iptables status

关闭防火墙
service iptables stop
chkconfig iptables off

查看防火墙状态
chkconfig iptables --list

4、SELINUX关闭

setenforce 0
vi /etc/selinux/config (SELINUX=disabled)

5、安装JDK并配置环境变量

5.1、使用rpm命令安装jdk

[root@cdh1 software]# rpm -ivh jdk-7u67-linux-x64.rpm

[root@cdh1 ~]# rpm -qa | grep jdk
[root@cdh1 ~]# rpm -ql jdk

5.2、配置jdk环境变量

[root@cdh1 ~]# vi .bash_profile

export JAVA_HOME=/usr/java/jdk1.7.0_67
export PATH=$PATH:$JAVA_HOME/bin

[root@cdh1 ~]# source .bash_profile

6、安装NTP

设置开机启动

[root@cdh1 ~]# chkconfig ntpd on

设置时间同步

[root@cdh1 ~]# ntpdate -u ntp.sjtu.edu.cn

7、安装并配置mysql

三台机器都进行mysql安装配置

7.1、安装mysql客户端

[root@cdh1 ~]# yum -y install mysql
[root@cdh2 ~]# yum -y install mysql
[root@cdh3 ~]# yum -y install mysql

查看mysql是否安装成功

[root@cdh1 ~]# rpm -ql mysql

7.2、安装mysql服务端

[root@cdh1 ~]# yum -y install mysql-server
[root@cdh2 ~]# yum -y install mysql-server
[root@cdh3 ~]# yum -y install mysql-server

[root@cdh1 ~]# yum -y install mysql-devel
[root@cdh2 ~]# yum -y install mysql-devel
[root@cdh3 ~]# yum -y install mysql-devel

7.3、配置mysql编码格式

[root@cdh1 ~]# vi /etc/my.cnf
[root@cdh2 ~]# vi /etc/my.cnf
[root@cdh3 ~]# vi /etc/my.cnf

添加

default-character-set=utf8

7.4、添加开机启动项

[root@cdh1 ~]# chkconfig --add mysqld
[root@cdh1 ~]# chkconfig mysqld on

[root@cdh2 ~]# chkconfig --add mysqld
[root@cdh2 ~]# chkconfig mysqld on

[root@cdh3 ~]# chkconfig --add mysqld
[root@cdh3 ~]# chkconfig mysqld on

7.5、启动mysql

[root@cdh1 ~]# service mysqld start
[root@cdh2 ~]# service mysqld start
[root@cdh3 ~]# service mysqld start

7.6、配置root用户并设置密码

cdh1主机、cdh2主机、cdh3主机，三台主机的mysql均进行如下配置

[root@master ~]# mysql -uroot -p
mysql> use mysql;
mysql> select user,host from user;
mysql> delete  from user where user = "";
mysql> select user,host from user;
mysql> update user set password = PASSWORD('123456') where user = 'root';
mysql> update user set host='localhost' where host='%';
mysql> select user,host,password from user;

7.7、赋予权限

mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
mysql> flush privileges;

8、配置本地yum源

9、下载第三方依赖包

[root@cdh1 ~]# yum -y install chkconfig  python  bind-utils  psmisc  libxslt  zlib  sqlite
[root@cdh1 ~]# yum -y install cyrus-sasl-gssapi  fuse  fuse-libs  redhat-lsb

Cloudera Manager安装

1、安装Cloudera Manager Server、 Agent

[root@cdh1 ~]# mkdir /opt/cloudera-manager
[root@cdh1 ~]# tar -zvxf cloudera-manager*.tar.gz -C /opt/cloudera-manager

[root@cdh2 ~]# mkdir /opt/cloudera-manager
[root@cdh2 ~]# tar -zvxf cloudera-manager*.tar.gz -C /opt/cloudera-manager

[root@cdh3 ~]# mkdir /opt/cloudera-manager
[root@cdh3 ~]# tar -zvxf cloudera-manager*.tar.gz -C /opt/cloudera-manager

2、创建用户cloudera-scm

[root@cdh1 ~]# useradd --system --home=/opt/cloudera-manager/cm-5.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

[root@cdh2 ~]# useradd --system --home=/opt/cloudera-manager/cm-5.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

[root@cdh3 ~]# useradd --system --home=/opt/cloudera-manager/cm-5.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

3、配置CM Agent

修改文件/opt/cloudera-manager/cm-5.4.3/etc/cloudera-scm-agent/config.ini中

[root@cdh1 ~]# vi /opt/cloudera-manager/cm-5.4.3/etc/cloudera-scm-agent/config.ini

server_host=cdh1

[root@cdh2 cloudera-manager]# vi /opt/cloudera-manager/cm-5.4.3/etc/cloudera-scm-agent/config.ini

server_host=cdh1

[root@cdh3 cloudera-manager]# vi /opt/cloudera-manager/cm-5.4.3/etc/cloudera-scm-agent/config.ini

server_host=cdh1

4、配置CM Server数据库

拷贝mysql jar文件到目录 /usr/share/java/

注意jar包名称要修改为mysql-connector-java.jar

[root@cdh1 software]# mkdir -p /usr/share/java/
[root@cdh1 ~]# mv mysql-connector-java-5.1.26-bin.jar mysql-connector-java.jar
[root@cdh1 software]# mv mysql-connector-java.jar /usr/share/java/
[root@cdh1 software]# cd /usr/share/java/

[root@cdh1 java]# mysql -uroot -p
mysql> grant all on *.* to 'temp'@'%' identified by 'temp' with grant option;
mysql> flush privileges;

[root@cdh1 java]# cd /opt/cloudera-manager/cm-5.4.3/share/cmf/schema/
[root@cdh1 schema]# ./scm_prepare_database.sh mysql temp -h cdh1 -utemp -ptemp --scm-host cdh1 scm scm scm

格式：数据库类型、数据库、数据库服务器、用户名、密码、 cm server服务器

5、创建Parcel目录

Server节点

[root@cdh1 schema]# mkdir -p /opt/cloudera/parcel-repo
[root@cdh1 schema]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo

Agent节点

[root@cdh1 init.d]# mkdir -p /opt/cloudera/parcels
[root@cdh1 init.d]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

[root@cdh2 cloudera-manager]# mkdir -p /opt/cloudera/parcels
[root@cdh2 cloudera-manager]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

[root@cdh3 cloudera-manager]# mkdir -p /opt/cloudera/parcels
[root@cdh3 cloudera-manager]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

6、制作CDH本地源

下载好文件CDH-5.4.0-1.cdh5.4.0.p0.27-el6.parcel以及manifest.json，

将这两个文件放到server节点的/opt/cloudera/parcel-repo下。

[root@cdh1 schema]# cd /opt/cloudera/parcel-repo

打开manifest.json文件，里面是json格式的配置，找到与下载版本相对应的hash码，

新建文件，文件名与你的parel包名一致，并加上.sha后缀，将hash码复制到文件中保存。

7、启动CM Server、 Agent

启动CM Server

做这个操作的时候，建立个快照

[root@cdh1 parcel-repo]# cd /opt/cloudera-manager/cm-5.4.3/etc/init.d/
[root@cdh1 init.d]# ./cloudera-scm-server start

cd /opt/cloudera-manager/cm-5.4.3/log/cloudera-scm-server
[root@cdh1 cloudera-scm-server]# tail -f cloudera-scm-server.log

Sever首次启动会自动创建表以及数据，不要立即关闭或重启，否则需要删除所有表及数据重新安装

启动CM Agent

[root@cdh1 ~]# cd /opt/cloudera-manager/cm-5.4.3/etc/init.d/
[root@cdh1 init.d]# ./cloudera-scm-agent start

[root@cdh2 cloudera-manager]# cd /opt/cloudera-manager/cm-5.4.3/etc/init.d/
[root@cdh2 init.d]# ./cloudera-scm-agent start

[root@cdh3 cloudera-manager]# cd /opt/cloudera-manager/cm-5.4.3/etc/init.d/
[root@cdh3 init.d]# ./cloudera-scm-agent start

8、访问：http://ManagerHost(主节点Ip):7180

在浏览器上输入：http://192.168.230.100:7180/cmf/login

若可以访问，则CM安装成功

用户、密码：admin

cat /proc/sys/vm/swappiness

echo 0 > /proc/sys/vm/swappiness

日志文件存放的位置

server和angent启动的日志
cd /opt/cloudera-manager/cm-5.4.3/log

安装服务的存放目录
cd /var/log

安装服务的配置文件存放目录
cd /etc/