CDH 5.8 installation

因工作需要，最近做了安装CDH,做了测试

Install cdh5.8.0(Centos6.5)

#step1
vi /etc/hosts
172.16.184.11 sp1.hadoop.com sp1
172.16.184.22 sp2.hadoop.com sp2
172.16.184.33 sp3.hadoop.com sp3
172.16.184.44 sp4.hadoop.com sp4
172.16.184.55 sp5.hadoop.com sp5
#step2
service iptables stop
chkconfig iptables off
#step3
sed -i's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
#ntpd
这里选择 sp1 节点为时钟同步服务器，其他节点为客户端同步时间到该节点。、

安装ntp:

$ yum install ntp

修改 cdh1 上的配置文件 /etc/ntp.conf :

restrict default ignore //默认不允许修改或者查询ntp,并且不接收特殊封包
restrict 127.0.0.1 //给于本机所有权限
restrict 172.16.184.0 mask 255.255.255.0 notrap nomodify //给于局域网机的机器有同步时间的权限
server 127.127.1.0 # local clock
driftfile /var/lib/ntp/drift
fudge 127.127.1.0 stratum 10

启动 ntp：

$ service ntpd start

设置开机启动:

$ chkconfig ntpd on

ntpq用来监视ntpd操作，使用标准的NTP模式6控制消息模式，并与NTP服务器通信。

ntpq -p 查询网络中的NTP服务器，同时显示客户端和每个服务器的关系。

$ ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*LOCAL(1) .LOCL. 5 l 6 64 1 0.000 0.000 0.000

"* "：响应的NTP服务器和最精确的服务器。
"+"：响应这个查询请求的NTP服务器。
"blank（空格）"：没有响应的NTP服务器。
"remote" ：响应这个请求的NTP服务器的名称。
"refid "：NTP服务器使用的更高一级服务器的名称。
"st"：正在响应请求的NTP服务器的级别。
"when"：上一次成功请求之后到现在的秒数。
"poll"：当前的请求的时钟间隔的秒数。
"offset"：主机通过NTP时钟同步与所同步时间源的时间偏移量，单位为毫秒（ms）。

客户端的配置

在cdh2和cdh3节点上执行下面操作：

$ ntpdate sp1

Ntpd启动的时候通常需要一段时间大概5分钟进行时间同步，所以在ntpd刚刚启动的时候还不能正常提供时钟服务，报错"no server suitable for synchronization found"。启动时候需要等待5分钟。

如果想定时进行时间校准，可以使用crond服务来定时执行。

00 1 * * * root /usr/sbin/ntpdate 172.16.184.11 >> /root/ntpdate.log 2>&1

这样，每天 1:00 Linux 系统就会自动的进行网络时间校准。

#step4(初始化网络)
/etc/init.d/network restart
#step5 (change transparents_hugepage)
echo never > /sys/kernel/mm/transparent_hugepage/enabled

vim /etc/sysctl.conf
vm.swappiness=0
sudo sysctl vm.swappiness=0
#step5
yum install createrepo
yuminstall httpd
service httpd start
chkconfig httpd on

#step6（安装外部数据库，这里选择mysql，默认为psql）

rpm -ivh mysql
chkconfig mysql on
service mysql on

/usr/bin/mysql_secure_installation
YYnYY

***create mewastore**
mysql -u root --password=111111 -e "create database metastore default character set utf8;"
mysql -u root --password=111111 -e "CREATE USER 'hive'@'%'IDENTIFIED BY '111111';GRANT ALL PRIVILEGES ON metastore.*TO'hive'@'%';FLUSH PRIVILEGES;"

mysql -u root --password=111111 -e "CREATE USER 'amon'@'%'IDENTIFIED BY '111111';"
mysql -u root --password=111111 -e "create database amon default character set utf8"
mysql -u root --password=111111 -e "GRANT ALL PRIVILEGES ON amon.*to'amon'@'%'"

mysql -u root --password=111111 -e "CREATE USER 'rman'@'%'IDENTIFIED BY '111111';"
mysql -u root --password=111111 -e "create database rman default character set utf8"
mysql -u root --password=111111 -e "GRANT ALL PRIVILEGES ON rman.*to'rman'@'%'"

mysql -u root --password=111111 -e "CREATE USER 'sentry'@'%'IDENTIFIED BY '111111';"
mysql -u root --password=111111 -e "create database sentry default character set utf8"
mysql -u root --password=111111 -e "GRANT ALL PRIVILEGES ON sentry.*to'sentry'@'%'"

mysql -u root --password=111111 -e "CREATE USER 'nav'@'%'IDENTIFIED BY '111111';"
mysql -u root --password=111111 -e "create database nav default character set utf8"
mysql -u root --password=111111 -e "GRANT ALL PRIVILEGES ON nav.*to'nav'@'%'"

mysql -u root --password=111111 -e "CREATE USER 'navms'@'%'IDENTIFIED BY '111111';"
mysql -u root --password=111111 -e "create database navms default character set utf8"
mysql -u root --password=111111 -e "GRANT ALL PRIVILEGES ON navms.*to'navms'@'%'"

mysql -u root --password=111111 -e "CREATE USER 'oozie'@'%'IDENTIFIED BY '111111';"
mysql -u root --password=111111 -e "create database oozie default character set utf8"
mysql -u root --password=111111 -e "GRANT ALL PRIVILEGES ON oozie.*to'oozie'@'%'"

mysql -u root --password=111111 -e "CREATE USER 'hue'@'%'IDENTIFIED BY '111111';"
mysql -u root --password=111111 -e "create database hue default character set utf8"
mysql -u root --password=111111 -e "GRANT ALL PRIVILEGES ON hue.*to'hue'@'%'"

mysql -u root --password=111111 -e "CREATE USER 'cm'@'%'IDENTIFIED BY '111111';"

mysql -u root --password=111111 -e "create database cm default character set utf8"
mysql -u root --password=111111 -e "GRANT ALL PRIVILEGES ON cm.*to'cm'@'%'"

step7 CDH5.8.0 install

mkdir cm
## take packages of CDH5.8.0 to cm
createrepo.
mkdir -p /var/www/html/cdh5/parcels/5.8.0/
mv manifest.json /var/www/html/cdh5/parcels/5.8.0/
mv cm /var/www/html/
chmod -R ugo+rx /var/www/html

step8 (create repo for yun install )
vi /etc/yum.repos.d/cdh.repo
[cdh]
name=cdh
baseurl=http://172.16.184.11/cm
enabled=true
gpgcheck=false

step9(install cloudera manager)
yum clean all

1 yum install oracle-j2sdk1.7
2 yum inatall cloudera-manager-daemons cloudera-manager-server

cp mysql-connector-java-5.15.jar /usr/share/java
ln -s mysql-connector-java-5.15.jar mysql-connector-java.jar

step10 (install external database)
/usr/share/cmf/schema/prepare_database.sh -h localhost mysql cm cm 111111
###
/usr/share/cmf/schema_prepare_database.sh -h <Mysqlhost> <DB_type> <DAtabase> <Username> <passwd>
<DB_type> can be mysql or oracle
DAtabase is cloudera manager database (cm)
username is cm
passwd is 111111

Step 11
service cloudera-scm-server start

listen port :7180

rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64
rpm -e --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64
rpm -e --nodeps libvirt-java-devel-0.4.9-1.el6.noarch
rpm -e --nodeps libvirt-java-0.4.9-1.el6.noarch
rpm -e --nodeps pki-java-tools-9.0.3-32.el6.noarch
rpm -e --nodeps java_cup-0.10k-5.el6.x86_64
rpm -e --nodeps java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64

遇到错误
1、Public key for postgresql-libs-8.4.18-1.el6_4.x86_64.rpm is not installed
解决：rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
yum install postgresql-libs-8.4.18-1.el6_4.x86_64
2、 MySQL-python不匹配
解决：镜像做 yum 源
yum install MySQL-python

3 spark gateway 不适用(正常现象)

4 权限问题

sudo -u hdfs hdfs dfs -chmod 1777 /tmp/logs
sudo -u hdfs hdfs dfs -chown mapred:hadoop /tmp/logs
sudo -u hdfs hdfs dfs -chgrp -R hadoop /tmp/logs

重启historyserver