服务器：
user: Administrator
password : Atide@1906
IP : 192.168.0.107
centos :
root
Atide@1906
虚拟机：
外网 内网
Hadoop001 Master root Atide@1906 192.168.0.112 192.168.100.1
Hadoop002 Slave root Atide@1906 192.168.0.113 192.168.100.2
Hadoop003 Slave root Atide@1906 192.168.0.114
Hadoop004 Slave root Atide@1906 192.168.0.115
Hadoop005 Slave root Atide@1906 192.168.0.116
Hadoop006 Slave root Atide@1906 192.168.0.117
安装：
1. 关闭防火墙
systemctl stop firewalld.service
systemctl disable firewalld.service
vi /etc/selinux/config
SELINUX=enforcing改为SELINUX=disabled
2. 修改主机名
vim /etc/hostname
hadoop001
//或hostnamectl --static set-hostname hadoop001
3. 修改hosts
vim /etc/hosts
192.168.0.112 hadoop001
4. 时间同步 http://www.centoscn.com/CentOS/config/2015/1105/6385.html
设置为上海时间
cp -f /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
#reboot
rpm –q ntp
yum –y install ntp
systemctl enable ntpd
systemctl start ntpd
ntpdate -u cn.pool.ntp.org
服务器：
# rpm -q ntp
# yum -y install ntp
systemctl enable ntpd
systemctl start ntpd
ntpdate -u cn.pool.ntp.org
vi /etc/ntp.conf
# Hosts on local network are less restricted.
# 允许内网其他机器同步时间
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
# 中国这边最活跃的时间服务器 : http://www.pool.ntp.org/zone/cn
server 210.72.145.44 perfer # 中国国家受时中心
server 202.112.10.36 # 1.cn.pool.ntp.org
server 59.124.196.83 # 0.asia.pool.ntp.org
# allow update time by the upper server
# 允许上层时间服务器主动修改本机时间
restrict 210.72.145.44 nomodify notrap noquery
restrict 202.112.10.36 nomodify notrap noquery
restrict 59.124.196.83 nomodify notrap noquery
# 外部时间服务器不可用时，以本地时间作为时间服务
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
修改完成后重启ntpd服务systemctl restart ntpd
使用ntpq -p 查看网络中的NTP服务器，同时显示客户端和每个服务器的关系
使用ntpstat 命令查看时间同步状态，这个一般需要5-10分钟后才能成功连接和同步。所以，服务器启动后需要稍等下：
刚启动的时候，一般是：
# ntpstat
unsynchronised
time server re-starting
polling server every 64 s
# ntpstat
synchronised to NTP server (202.112.10.36) at stratum 3
time correct to within 275 ms
polling server every 256 s
其他：
vim /etc/ntp.conf
driftfile /var/lib/ntp/drift
restrict 127.0.0.1
restrict -6 ::1
# 配置时间服务器为本地的时间服务器
server 192.168.100.1
restrict 192.168.100.1 nomodify notrap noquery
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
# ntpdate -u 192.168.100.1
// 卸载ntp
rpm -qa | grep ntp
rpm -e --nodeps ntp-4.2.6p5-25.el7.centos.2.x86_64
rpm -e --nodeps ntpdate-4.2.6p5-25.el7.centos.2.x86_64
5. 免密登录
ssh-keygen
输入password
ssh-copy-id hadoop002
软件安装
参考：http://www.cnblogs.com/jasondan/p/4011153.html
1.卸载openJDK
2.安装JDK1.8 安装位置必须为这，否则找不到
mkdir -p /usr/java
tar -zxvf jdk-8u121-linux-x64.tar.gz -C /usr/java/
cd /usr/java/jdk1.8.0_121
vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_121
exportCLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
source /etc/profile
scp -r /usr/java root@hadoop002:/usr/
cat << EOF >> /etc/profile
JAVA_HOME=/usr/java/jdk1.8.0_121
PATH=\$JAVA_HOME/bin:\$PATH:.
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME
export PATH
export CLASSPATH
EOF
source /etc/profile
3.安装Mysql
yum remove mariadb-libs
rpm –ivh mysql-community-common-5.7.18-1.e17.x86_64.rpm
rpm –ivh mysql-community-libs-5.7.18-1.e17.x86_64.rpm
rpm –ivh mysql-community-client-5.7.18-1.e17.x86_64.rpm
rpm –ivh mysql-community-server-5.7.18-1.e17.x86_64.rpm
配置MySQL，vi /etc/my.cnf在文件最后添加skip-grant-tables
systemctl start mysqld
update mysql.user set authentication_string=password('123456') where User='root' and Host='localhost';
flush privileges;
#hive
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
#activity monitor
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
grant all privileges on *.* to 'root'@'hadoop001' identified by '123456' with grant option;
flush privileges;
创建mysql数据库
1 . –hive 数据库
mysql> create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
2 . –集群监控数据库 amon
mysql> create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
3 . –hue 数据库
mysql> create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
4 . –oozie 数据库
mysql> create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
6）为新创建的库赋权限
如果对mysql数据库权限要求比较高的话可以手动单独处理 如果要求不是很高，直接使用以下命令即可：
授权root用户在主节点(cloud032)拥有所有数据库的访问权限
mysql> grant all privileges on . to ‘root’@’cloud032’ identified by ‘root’ with grant option;
mysql> flush privileges;
4.安装CDM
在主节点初始化CM5的数据库：
tar -zxvf cloudera-manager-centos7-cm5.12.0_x86_64.tar.gz -C /opt
cp mysql-connector-java-5.1.32-bin.jar /opt/cm-5.12.0/share/cmf/lib/mysql-connector-java-5.1.32-bin.jar
cp mysql-connector-java-5.1.32-bin.jar /opt/cm-5.10.1/share/cmf/lib/mysql-connector-java-5.1.32-bin.jar
sh /opt/cm-5.12.0/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p scm scm
sh /opt/cm-5.10.1/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p scm scm
出现All done, your SCM database is configured correctly!表示安装成功！
如果报错：Caused by: java.net.ConnectException: Connection refused (Connection refused)
设置完帐号密码之后未重启mysql
如果报错：java.sql.SQLException: Your password does not satisfy the current policy requirements
>> mysql -uroot -p 登录数据库
mysql> select @@validate_password_policy;
mysql> set global validate_password_policy=0;
set global validate_password_length=4;
set PASSWORD=PASSWORD(‘123456’);
配置cm agent
>> cd /opt/cm-5.10.1/etc/cloudera-scm-agent
>> vim config.ini
CM Server的hostname: server_host=ha01.nicigo.com
log file的位置:log_file=/data/hadoop/logs/cloudera-scm-agent/cloudera-scm-agent.log
parcel_dir的位置:parcel_dir=/data/hadoop/cloudera/parcels
****jdbc依赖包的位置:cloudera_mysql_connector_jar=/opt/cm-5.12.0/share/cmf/lib/mysql-connector-java-5.1.32-bin.jar
复制cm-5.11.0包到所有节点，需要切换root用户
>> scp -r cm-5.12.0/ root@hadoop002:/opt/
在所有节点创建cloudera-scm用户
useradd --system --home=/opt/cm-5.10.1/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
Agent配置
vi /opt/cm-5.12.0/etc/cloudera-scm-agent/config.ini 修改servername
同步Agent到其他节点
scp -r /opt/cm-5.12.0 root@hadoop002:/opt/
准备Parcels，用以安装CDH5
将CHD5相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中（parcel-repo需要手动创建）。
相关的文件如下：
CDH-5.1.3-1.cdh5.1.3.p0.12-el6.parcel
CDH-5.1.3-1.cdh5.1.3.p0.12-el6.parcel.sha1
manifest.json
将CDH-5.1.3-1.cdh5.1.3.p0.12-el6.parcel.sha1，重命名为CDH- 5.1.3-1.cdh5.1.3.p0.12-el6.parcel.sha
相关启动脚本
通过/opt/cm-5.10.1/etc/init.d/cloudera-scm-server start启动服务端。
通过/opt/cm-5.10.1/etc/init.d/cloudera-scm-agent start启动Agent服务。
http://192.168.0.112:7180/cmf/login
pstree: command not found
yum install -y psmisc
创建存Hadoop各组件元数据的库
-- 创建Hive数据元数据库，并赋予权限
create database metastore DEFAULT CHARACTER SET latin1;
grant all on metastore.* TO 'hive'@'%'IDENTIFIED BY '123456';
-- 创建scm数据库，并赋予权限
create database scm DEFAULT CHARACTER SET utf8;
grant all on scm.* TO 'scm'@'%' IDENTIFIED BY '123456';
-- 创建amon数据库，并赋予权限
create database amon DEFAULT CHARACTER SET utf8;
grant all on amon.* TO 'amon'@'%'IDENTIFIED BY '123456';
-- 创建rmon数据库，并赋予权限 reports manager
create database rmon DEFAULT CHARACTER SET utf8;
grant all on rmon.* TO 'scm'@'%'IDENTIFIED BY '123456';
-- 创建sentry数据库，并赋予权限
create database sentry DEFAULT CHARACTER SET utf8;
grant all on sentry.* TO 'sentry'@'%'IDENTIFIED BY '123456';
-- 创建hue数据库，并赋予权限
create database hue CHARACTER SET utf8;
grant all on hue.* to 'hue'@'%' identified by '123456';
-- 创建oozie数据库
create database oozie CHARACTER SET utf8;
grant all privileges on oozie.* to'oozie'@'%' identified by '123456';
待。。。
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
grant all on *.* to root@'%' Identified by '123456'
//grant all privileges on *.* to 'root'@'主机名' identified by '密码' with grant option;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION;
drop database hue
flush privileges;
问题：
1.Cloudera 建议将 /proc/sys/vm/swappiness 设置为 0。当前设置为 60。使用 sysctl 命令在运行时更改该设置并编辑 /etc/sysctl.conf 以在重启后保存该设置。您可以继续进行安装，但可能会遇到问题，Cloudera Manager 报告您的主机由于交换运行状况不佳。以下主机受到影响：
echo 0 > /proc/sys/vm/swappiness
2.已启用透明大页面压缩，可能会导致重大性能问题。请运行“echo never > /sys/kernel/mm/transparent_hugepage/defrag”和“echo never > /sys/kernel/mm/transparent_hugepage/enabled”以禁用此设置，然后将同一命令添加到 /etc/rc.local 等初始化脚本中，以便在系统重启时予以设置。
//echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag' >> /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
每台受影响的主机运行命令
echo never > /sys/kernel/mm/transparent_hugepage/defrag
并且
vim /etc/rc.local
加入echo never > /sys/kernel/mm/transparent_hugepage/defrag这条命令。
然后点击重新运行进行重新检查
yum install -y python-lxml
网上找了很多文章，貌似只有在CDH5.8.3的版本才开始出现的，由于我本身是使用二进制安装的mysql5.7的数据库，CentOS6.5中原来自带的mysql的rpm包我都删除了，所以找不到。我从其他系统相同的服务器中拷贝了一个libmysqlclient_r.so.16 ，正常位于/usr/lib64/mysql目录下，把这个包拷贝到本机的/usr/lib64下即可，随后执行ldconfig，随后测试hue数据库连接OK
cp /mnt/hgfs/tools/MySQL/mysql-connector-java-5.1.36-bin.jar /opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/oozie/libext
cp mysql-connector-java-5.1.32-bin.jar /opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/oozie/libext
继续安装，启动hive的时候失败，看日志是没有Mysql的JDBC，cp一个过去再继续
cp mysql-connector-java-5.1.40-bin.jar /opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hive/lib
安装oozie的时候失败，还是JDBC，再cp一个
cp mysql-connector-java-5.1.40-bin.jar /var/lib/oozie/
http://blog.csdn.net/shushugood/article/details/76686062 ldconfig
/usr/lib64/ 目录下 应为18版
我从其他系统相同的服务器中拷贝了一个libmysqlclient_r.so.16 ，正常位于/usr/lib64/mysql目录下，把这个包拷贝到本机的/usr/lib64下即可，随后执行ldconfig，随后测试hue数据库连接OK
日志：/opt/cm-5.10.1/run/cloudera-scm-agent/process/30-HUE-test-db-connection/logs
http://192.168.0.109:7180/cmf/license/wizard?returnUrl=%2Fcmf%2Fexpress-wizard%2Fwelcome#step=selectLicenseStep
扫一扫
1655
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
