cdh mysql5.7_CDH5.7.2离线部署笔记

#设置root授权访问以上所有的数据库：

grant all privileges on *.* to 'root'@'dcnn1' identified by 'bigdata' with grant option;

flush privileges;

#授权用户在服务端拥有数据库的访问权限

grant all privileges on *.* to 'oozie'@'dcnn1 ' identified by 'oozie' with grant option;

grant all privileges on *.* to 'hive'@'dcnn1' identified by 'hive' with grant option;

grant all privileges on *.* to 'amon'@'dcnn1 ' identified by 'amon' with grant option;

grant all privileges on *.* to 'report'@'dcnn1 ' identified by 'report' with grant option;

flush privileges;

#授权root用户在客户端拥有数据库的访问权限

GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' IDENTIFIED BY 'hive' WITH GRANT OPTION;

GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%' IDENTIFIED BY 'oozie' WITH GRANT OPTION;

GRANT ALL PRIVILEGES ON *.* TO 'amon'@'%' IDENTIFIED BY 'amon' WITH GRANT OPTION;

GRANT ALL PRIVILEGES ON *.* TO 'report'@'%' IDENTIFIED BY 'report' WITH GRANT OPTION;

flush privileges;

2.9安装cm

cm和namenode复用了，机器太少了

1)将下载的cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz上传至dcnn1目录/opt下(该目录为Cloudera manager的默认目录)

2)tar xvfz cloudera-manager*.tar.gz，解压后生成两个目录cloudera和cm-5.7.2

3)所有节点创建用户cloudera-scm

sudo useradd --system --home=/opt/cm-5.7.2/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

4)为Cloudera Manager 5建立数据库

首先需要去MySql的官网下载JDBC驱动mysql-connector-java-5.1.38.tar.gz(可以下载最新的版本)，下载地址：http://dev.mysql.com/downloads/connector/j/ ，解压后找到mysql-connector-java-5.1.38-bin.jar放到/opt/cm-5.7.2/share/cmf/lib/中： cp mysql-connector-java-5.1.38-bin.jar /opt/cm-5.7.2/share/cmf/lib/

/opt/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql cm -h dcnn1 -uroot -pbigdata --scm-host dcnn1 scm scm scm

遇到问题：未解决，但不影响后面得部署

AVA_HOME=/usr/java/jdk1.8.0_60

Verifying that we can write to /opt/cm-5.7.2/etc/cloudera-scm-server

log4j:ERROR Could not find value for key log4j.appender.A

log4j:ERROR Could not instantiate appender named "A".

Creating SCM configuration file in /opt/cm-5.7.2/etc/cloudera-scm-server

Executing:  /usr/java/jdk1.8.0_60/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.7.2/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cm-5.7.2/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.

log4j:ERROR Could not find value for key log4j.appender.A

log4j:ERROR Could not instantiate appender named "A".

agent配置

1)修改/opt/cm-5.7.2/etc/cloudera-scm-agent/config.ini中的server_host， server_host为主节点的主机名

2)同步Agent到其他节点

sudo scp -r /opt/cm-5.7.2 root@dcnn2:/opt/;

sudo scp -r /opt/cm-5.7.2 root@dcdn01:/opt/;

sudo scp -r /opt/cm-5.7.2 root@dcdn02:/opt/;

sudo scp -r /opt/cm-5.7.2 root@dcdn03:/opt/;

2.10启动服务

通过sudo /opt/cm-5.7.2/etc/init.d/cloudera-scm-server start 启动服务端

通过sudo /opt/cm-5.7.2/etc/init.d/cloudera-scm-agent start 启动agent端

我们启动的其实是个service脚本，需要停止服务将以上的start参数改为stop就可以了，重启是restart。

2.11 通过cm部署CDH

1)登录http://10.20.20.1:7180

2)根据向导就可以一步一步进行了，比较简单。

3 遇到问题

1、hive和oozie启动失败

缺少JDBC驱动，因为我的mysql装在远端，这里要访问的时候要通过JDBC驱动。因此将mysql-connector-java-5.1.22-bin.jar添加到hive和oozie的lib：

sudo cp mysql-connector-java-5.1.38-bin.jar /opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hive/lib

sudo cp mysql-connector-java-5.1.38-bin.jar /opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/oozie/libext

2、安装后hive在查询时失败，原因是权限问题

org.apache.hadoop.security.AccessControlException: Permission denied: user=bigdata, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x

最简单的处理办法是：服务范围勾选去掉。