Cloudera Manager离线安装

最新推荐文章于 2020-11-28 16:06:45 发布

qq_32640793

最新推荐文章于 2020-11-28 16:06:45 发布

阅读量707

点赞数

文章标签： CDH Cloudera Manager离线安装

本文链接：https://blog.csdn.net/qq_32640793/article/details/82499046

版权

环境：centos7
一、所需安装包
JDK包
JDK版本：jdk-8u181-linux-x64.tar.gz
CM包
CM版本：5.10.0 cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz
下载地址：http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz
CDH包
CDH版本：5.10.0
CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel
CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha1 manifest.json
manifest.json
下载地址
http://archive.cloudera.com/cdh5/parcels/5.10.0/CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel
http://archive.cloudera.com/cdh5/parcels/5.10.0/CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha1
http://archive.cloudera.com/cdh5/parcels/5.10.0/manifest.json
java-mysql连接jar包
下载地址：http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.22/mysql-connector-java-5.1.22.jar

节点
192.168.1.124 n1
192.168.1.125 n2
192.168.1.126 n3
二、环境准备
1、静态ip配置(所有节点)
vi /etc/sysconfig/network-scripts/ifcfg-e（按Tab键）

修改：
BOOTPROTO=“static”

BROADCAST=192.168.1.255
IPADDR=192.168.1.124
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=192.168.1.1
重启网络：service network restart
2. 网络配置(所有节点)
vi /etc/hosts

192.168.1.124 n1
192.168.1.125 n2
192.168.1.126 n3

vi /etc/sysconfig/network 修改hostname：

NETWORKING=yes
HOSTNAME=n1
别名设置（可选）：sudo hostnamectl set-hostname
重启网络：service network restart

3、关闭防火墙(所有节点)

systemctl stop firewalld.service
systemctl disable firewalld.service
查看状态：firewall-cmd --state

4、关闭SELinux(所有节点)

vi /etc/selinux/config
SELINUX=disabled
查看状态：getenforce

5、reboot
6、根目录创建softwore文件夹（所有的安装包都放入）
7、安装jdk(所有节点)

yum remove openjdk
rpm -e --nodeps tzdata-java-2014b-1.el7.noarch
tar –zxvf jdk-8u181-linux-x64.tar.gz

vi /etc/profile 添加

export JAVA_HOME=/softwore/jdk1.8.0_181
export JRE_HOME= ${JAVA_HOME}/jre export CLASSPATH=.:$ {JAVA_HOME}/lib: ${JRE_HOME}/lib export PATH=$ {JAVA_HOME}/bin:$PATH

配合jdk成功后有时很多命令用不成了执行：
export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

8、时间同步

主节点： vi /etc/ntp.conf

找到restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap，取消注释
找到server 0.centos.pool.ntp.org iburst，并将所有server配置进行注释
添加：
server 127.127.1.0
fudge 127.127.1.0 stratum 10

每个从节点：
vi /etc/ntp.conf
添加：
server n1(主节点名或ip）

所有节点：
systemctl restart ntpd
chkconfig ntpd on

9、单向免密(所有节点)

ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
scp ~/.ssh/authorized_keys root@ slave1:~/.ssh/
scp ~/.ssh/authorized_keys root@ slave2:~/.ssh/

10、mysql安装（主节点）

find / -name libmysqlclient.so.18
找到libmysqlclient.so.18文件拷出来（为装hue做准备）

rpm -qa | grep -i mysql
rpm -ev --nodeps

find / -name mysql
rm -rf /var/log/mysqld.log

删除mariadb：
rpm -qa|grep mariadb
rpm -e --nodeps

rpm -ivh mysql-community-common-5.7.18-1.el6.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.18-1.el6.x86_64.rpm
rpm -ivh mysql-community-libs-compat-5.7.18-1.el6.x86_64.rpm
rpm -ivh mysql-community-client-5.7.18-1.el6.x86_64.rpm
rpm -ivh mysql-community-server-5.7.18-1.el6.x86_64.rpm
rpm -ivh mysql-community-devel-5.7.18-1.el6.x86_64.rpm

chkconfig mysqld on
service mysqld start
grep ‘temporary password’ /var/log/mysqld.log
mysql -u root –p
初始密码
set global validate_password_policy=0;
set global validate_password_length=0;
设置密码：
ALTER USER ‘root’@‘localhost’ IDENTIFIED BY ‘Pwd@123456’;
flush privileges;
quit

mysql -u root -p
Pwd@123456

create database if not exists amon default charset utf8 collate utf8_general_ci;
create database if not exists rman default charset utf8 collate utf8_general_ci;
create database if not exists nav default charset utf8 collate utf8_general_ci;
create database if not exists navms default charset utf8 collate utf8_general_ci;
create database if not exists hue default charset utf8 collate utf8_general_ci;
create database if not exists sentry default charset utf8 collate utf8_general_ci;
create database if not exists hive;
create database if not exists oozie;
grant all on hive.* to ‘hive’@’%’ identified by ‘Pwt@123456’ with grant option;
grant all on oozie.* to ‘oozie’@’%’ identified by ‘Pwt@123456’ with grant option;
grant all on hue.* to ‘hue’@’%’ identified by ‘Pwt@123456’ with grant option;
grant all on amon.* to ‘amon’@’%’ identified by ‘Pwt@123456’ with grant option;
grant all on rman.* to ‘rman’@’%’ identified by ‘Pwt@123456’ with grant option;
grant all on nav.* to ‘nav’@’%’ identified by ‘Pwt@123456’ with grant option;
grant all on navms.* to ‘navms’@’%’ identified by ‘Pwt@123456’ with grant option;
grant all on sentry.* to ‘sentry’@’%’ identified by ‘Pwd@123456’ with grant option;
grant all privileges on . to ‘root’@‘n1’ identified by ’ Pwd@123456’ with grant option;
grant all privileges on . to ‘root’@’%’ identified by ‘Pwd@123456’ with grant option;
flush privileges;

yum安装时可能出现问题：yum install mysql-server 异常：No package mysql-server available.问题

解决：wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
rpm -ivh mysql-community-release-el7-5.noarch.rpm
yum install mysql-server 命令就可以成功安装了

三、正式开工
1、主节点解压安装

cloudera manager的目录默认位置在/opt下，解压：tar xzvf cloudera-manager*.tar.gz将解压后的cm-5.10.0和cloudera目录放到/opt目录下。
为Cloudera Manager 5建立数据库
将mysql-connector-java-5.1.22.jar，放到/opt/cm-5.10.0/share/cmf/lib/中。
在主节点初始化CM5的数据库：
/opt/cm-5.10.0/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p Pwd@123456 scm scm
Agent配置
修改/opt/cm-5.10.0/etc/cloudera-scm-agent/config.ini中的server_host为主节点的主机名。
同步Agent到其他节点
scp -r /opt/cm-5.10.0 root@ n2:/opt/
scp -r /opt/cm-5.10.0 root@ n3:/opt/
在所有节点创建cloudera-scm用户
useradd --system --home=/opt/cm-5.10.0/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment “Cloudera SCM User” cloudera-scm

2、准备Parcels，用以安装CDH5

将CHD5相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中
相关的文件如下：
CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel
CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha1
manifest.json
最后将CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha1，重命名为CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha
相关启动脚本
通过/opt/cm-5.10.0/etc/init.d/cloudera-scm-server start启动服务端。
通过/opt/cm-5.10.0/etc/init.d/cloudera-scm-agent start启动Agent服务。

开始部署
这里写图片描述
1、检查主机正确性时出现 “Cloudera 建议将 /proc/sys/vm/swappiness 设置为 0。当前设置为 30。” 的警告，进行如下设定
vi /etc/sysctl.conf
vm.swappiness = 0
sysctl –p
2、检查主机正确性时出现 “已启用“透明大页面”，它可能会导致重大的性能问题。” 的警告，进行如下设定
vi /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Hive安装问题

1、The specified datastore driver (“com.mysql.jdbc.Driver”) was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
解决：将mysql驱动包放到对应的服务器的安装目录下，再启动hive即可
cp mysql-connector-java-5.1.34.jar /opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/hive/lib/

2、Required table missing : “VERSION” in Catalog “” Schema “”. DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable “datanucleus.autoCreateTables”
org.datanucleus.store.rdbms.exceptions.MissingTableException: Required table missing : “VERSION” in Catalog “” Schema “”. DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable “datanucleus.autoCreateTables”
解决：如图设置
这里写图片描述

Kafka安装问题

1、下载这3个文件
这里写图片描述
2、将KAFKA-1.2.0.jar上传到/opt/cloudera/csd目录下
将KAFKA-2.2.0-1.2.2.0.p0.68-el7.parcel和KAFKA-2.2.0-1.2.2.0.p0.68-el7.parcel.sha上传到/opt/cloudera/parcel-repo/目录下
3、激活kafka配置

4、启动kafka报错：OutofMemeryException
5、设置Java Heap Size of Broker大小
这里写图片描述
6、重新启动ok

自带Spark安装

问题：java home不存在
find / -type f -name “*.sh” | xargs grep "as ALT_NAME"找到 /opt/cm-5.10.0/lib64/cmf/service/client/deploy-cc.sh文件

vi /opt/cm-5.10.0/lib64/cmf/service/client/deploy-cc.sh
在echo "using $JAVA_HOME as JAVA_HOME"行上面添加：

JAVA_HOME=/softwore/jdk1.8.0_181
export JAVA_HOME=/softwore/jdk1.8.0_181

然后发送其他节点

安装成功后检测：

验证方法1：
./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/spark/lib/spark-examples.jar 10

错误：Exception in thread “main” java.lang.IllegalArgumentException: Required executor memory (1024+384 MB) is above the max thresh old (1024 MB) of this cluster! Please check the values of ‘yarn.scheduler.maximum-allocation-mb’ and/or ‘yarn.nodemanager.resource.memory-mb’.
yarn.scheduler.maximum-allocation-mb 和 yarn.nodemanager.resource.memory-mb为2

错误：Exception in thread “main” org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
su hdfs
hadoop fs -chmod 777 /user

非自带spark安装
下载这4个文件
这里写图片描述
将SPARK2_ON_YARN-2.1.0.cloudera1.jar上传到/opt/cloudera/csd目录下（主节点）
cd /opt/cloudera/csd/
chown cloudera-scm:cloudera-scm SPARK2_ON_YARN-2.1.0.cloudera1.jar
chmod 644 SPARK2_ON_YARN-2.1.0.cloudera1.jar
重命名/opt/cloudera/parcel-repo/下的manifest.json（cdh）文件
将剩下3个文件上传到/opt/cloudera/parcel-repo/目录下

hue安装问题
1、libmysqlclient.so.18不能被发现（卸载系统自带的mysql时将这个文件卸载掉了）
这里写图片描述
jango.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: libmysqlclient.so.18: cannot open shared object file: No such file or directory
将先前copy出来的libmysqlclient.so.18（注意系统版本）文件放在/usr/lib64/下

ntp:https://blog.csdn.net/vic_qxz/article/details/80343351

qq_32640793

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Cloudera Manager离线安装

环境：centos7 一、所需安装包 JDK包 JDK版本：jdk-8u181-linux-x64.tar.gz CM包 CM版本：5.10.0 cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz 下载地址：http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.1...
复制链接

扫一扫