cloudera安装

软件下载地址：http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/

https://www.zybuluo.com/sasaki/note/242142

系统已经优化完毕

一、下载软件并安装

1、CDH下载

http://archive.cloudera.com/cm5/installer/

http://archive.cloudera.com/cm5/installer/5.6.0/cloudera-manager-installer.bin

2、修改主机名

vi /etc/sysconfig/network

3、修改hosts

192.168.202.34  csmaster

192.168.202.42  csslave01

192.168.202.43  csslave02

192.168.202.44  csslave03

192.168.202.53  csslave03

4、ssh免密登录

[root@csmaster ~]# mkdir ~/.ssh

[root@csmaster ~]# ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key(/root/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in/root/.ssh/id_rsa.

Your public key has been saved in/root/.ssh/id_rsa.pub.

The key fingerprint is:

a4:0a:35:26:d3:3e:df:64:30:c0:a6:05:04:2d:c7:24root@csmaster

The key's randomart image is:

+--[ RSA 2048]----+

|E*+o.            |

|..+.+.           |

| oo+= o .        |

|  .*. =         |

|  .o . S        |

|   .+ +         |

|   . . .        |

|                 |

|                 |

+-----------------+

[root@csmaster ~]# ll

total 24

-rw-r--r-- 1 root root  297 May 3 10:54 hostname.sh

-rw-r--r-- 1 root root 3251 May  3 10:54 localsource6.sh

-rw-r--r-- 1 root root 1321 May  3 10:54 makegood_1.sh

-rw-r--r-- 1 root root 5635 May  3 10:54 makegood_2.sh

-rw-r--r-- 1 root root 3024 May  3 10:54 makegood_3.sh

[root@csmaster ~]# ls -la

total 60

dr-xr-x---. 3 root root 4096 May 10 17:37 .

dr-xr-xr-x. 26 root root 4096 May 10 16:26..

-rw-------  1 root root  939 May 10 17:28.bash_history

-rw-r--r--. 1 root root   18 May 20  2009 .bash_logout

-rw-r--r--. 1 root root  176 May 20  2009 .bash_profile

-rw-r--r--. 1 root root  176 Sep 23  2004 .bashrc

-rw-r--r--. 1 root root  100 Sep 23  2004 .cshrc

-rw-r--r--  1 root root  297 May  3 10:54 hostname.sh

-rw-r--r--  1 root root 3251 May  3 10:54localsource6.sh

-rw-r--r--  1 root root 1321 May  3 10:54makegood_1.sh

-rw-r--r--  1 root root 5635 May  3 10:54makegood_2.sh

-rw-r--r--  1 root root 3024 May  3 10:54makegood_3.sh

drwxr-xr-x  2 root root 4096 May 10 17:37 .ssh

-rw-r--r--. 1 root root  129 Dec  4  2004.tcshrc

[root@csmaster ~]# cd .ssh

[root@csmaster .ssh]# ll

total 8

-rw------- 1 root root 1675 May 10 17:37id_rsa

-rw-r--r-- 1 root root  395 May 10 17:37 id_rsa.pub

[root@csmaster .ssh]# cat ~/.ssh/id_rsa.pub >> authorized_keys

[root@csmaster .ssh]# ll

total 12

-rw-r--r-- 1 root root  395 May 10 17:38 authorized_keys

-rw------- 1 root root 1675 May 10 17:37id_rsa

-rw-r--r-- 1 root root  395 May 10 17:37 id_rsa.pub

[root@csmaster .ssh]# chmod 600 ~/.ssh/authorized_keys

 

scp ~/.ssh/authorized_keyscsslave01:~/.ssh/

scp ~/.ssh/authorized_keyscsslave02:~/.ssh/

scp ~/.ssh/authorized_keyscsslave03:~/.ssh/

scp ~/.ssh/authorized_keyscsslave04:~/.ssh/

 [root@csmaster .ssh]# ssh csslave01

Warning: Permanently added'csslave01,192.168.202.42' (RSA) to the list of known hosts.

[root@csslave01 ~]# exit

logout

Connection to csslave01 closed.

[root@csmaster .ssh]# ssh csslave02

Warning: Permanently added'csslave02,192.168.202.43' (RSA) to the list of known hosts.

[root@csslave02 ~]# exit

logout

Connection to csslave02 closed.

[root@csmaster .ssh]# ssh csslave03

Warning: Permanently added'csslave03,192.168.202.44' (RSA) to the list of known hosts.

[root@csslave03 ~]# exit

logout

Connection to csslave03 closed.

[root@csmaster .ssh]# ssh csslave04

Warning: Permanently added'csslave04,192.168.202.53' (RSA) to the list of known hosts.

[root@csslave04 ~]# exit

logout

Connection to csslave04 closed.

5、安装jdk

yum install java-1.7.0-openjdk.x86_64 java-1.7.0-openjdk-devel.x86_64–y

 

6、关闭防火墙

service iptables stop #临时关闭

chkconfig iptables off #重启后生效

7、配置NTP服务

[root@i-waaeswlp ~]# crontab -e

no crontab for root - using an empty one

 

*/5 * * * * /usr/sbin/ntpdatentp.wumart.com >/dev/null 2>&1

8、优化

关闭反向解析

mv /usr/bin/host /usr/bin/host.bak

关闭swap

echo 0 > /proc/sys/vm/swappiness

 

这样操作重启机器还是还原，要永久改变

vim  /etc/sysctl.conf

vm.swappiness=0

 

关闭透明大页面压缩（以禁用此设置，然后将同一命令添加到 /etc/rc.local 等初始化脚本中，以便在系统重启时予以设置）

echo never >/sys/kernel/mm/transparent_hugepage/defrag

echo never >/sys/kernel/mm/transparent_hugepage/enabled

 

 

9、建立用户

# useradd wumart

wumart要设置密码：SmallDataWumart

#useradd spark2

spark2设置密码：SmallDataWumart

同样建立免密

[root@csmaster ~]# su - wumart

[wumart@csmaster ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key(/home/wumart/.ssh/id_rsa):

Created directory '/home/wumart/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in/home/wumart/.ssh/id_rsa.

Your public key has been saved in/home/wumart/.ssh/id_rsa.pub.

The key fingerprint is:

a8:aa:aa:08:91:a6:b6:9d:62:e2:11:fa:97:4d:6d:76wumart@csmaster

The key's randomart image is:

+--[ RSA 2048]----+

|                 |

|                 |

|                 |

| .    .         |

|oo   ..S        |

|+.. .. + E      |

|+o .+ o .       |

|=++.+ .          |

|@+++             |

+-----------------+

ssh-keygen -t rsa

ssh-copy-id -i ~/.ssh/id_rsa.pubwumart@csmaster

ssh-copy-id -i ~/.ssh/id_rsa.pubwumart@csslave01

ssh-copy-id -i ~/.ssh/id_rsa.pubwumart@csslave02

ssh-copy-id -i ~/.ssh/id_rsa.pubwumart@csslave03

ssh-copy-id -i ~/.ssh/id_rsa.pubwumart@csslave04

 

ssh-keygen -t rsa

ssh-copy-id -i ~/.ssh/id_rsa.pubspark2@csmaster

ssh-copy-id -i ~/.ssh/id_rsa.pubspark2@csslave01

ssh-copy-id -i ~/.ssh/id_rsa.pubspark2@csslave02

ssh-copy-id -i ~/.ssh/id_rsa.pubspark2@csslave03

ssh-copy-id -i ~/.ssh/id_rsa.pubspark2@csslave04

 

记得在每台客户机上更改.ssh目录权限

chmod 700 .ssh

文件权限

chmod 600 authorized_keys

二、安装Cloudera Manager Server和Agent

1、在master节点安装

#https://www.cloudera.com/documentation/enterprise/release-notes/topics/cm_vd.html各版本地址
#https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.6.0/RPMS/x86_64本次实验版本地址

 

安装方式1：下载所有包

http://blog.sina.com.cn/s/blog_b10920ac0102w8dw.html

下载所有包

yum localinstall –-nogpgcheck  *.rpm(安装rpm包)

 

客户端安装

yum localinstall –-nogpgcheck  *.rpm

 

service cloudera-scm-server-db start

service cloudera-scm-server start

service cloudera-scm-agent start

安装方式2：在线安装cm（国外网速比较慢，一般都失败）

cloudera-manager-installer.bin

[root@csmaster soft]#./cloudera-manager-installer.bin

安装方式3：下载安装（没有成功，网上建议用mysql）

下载软件

wgethttp://archive-primary.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.6.0_x86_64.tar.gz

wgethttp://archive-primary.cloudera.com/cdh5/parcels/5.6.0/CDH-5.6.0-1.cdh5.6.0.p0.45-el6.parcel

wgethttp://archive-primary.cloudera.com/cdh5/parcels/5.6.0/CDH-5.6.0-1.cdh5.6.0.p0.45-el6.parcel.sha1

wget http://archive-primary.cloudera.com/cdh5/parcels/5.6.0/manifest.json

mv CDH-5.6.0-1.cdh5.6.0.p0.45-el6.parcel.sha1CDH-5.6.0-1.cdh5.6.0.p0.45-el6.parcel.sha

 

创建用户

useradd --system--home=/opt/cm-5.6.0/run/cloudera-scm-server --no-create-home--shell=/bin/false --comment "Cloudera SCM User" cloudera-scm 

比较好的教材

http://azhuang.blog.51cto.com/9176790/1748432/

http://blog.csdn.net/gavin_rengf/article/details/51326824

https://www.zybuluo.com/sasaki/note/242142

 

2、网页继续安装

http://192.168.202.34:7180/

手工安装需要，更改vim/etc/cloudera-scm-agent/config.ini  server_host改成主节点

[root@csmaster.ssh]# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.131-2.6.9.0.el6_8.x86_64

[root@csmaster.ssh]# rpm -e --nodeps java-1.7.0-openjdk-devel-1.7.0.131-2.6.9.0.el6_8.x86_64

[root@csmaster.ssh]# rpm -e --nodeps tzdata-java-2017b-1.el6.noarch

[root@csmaster.ssh]# java version

wgetftp://10.249.100.249/pub/jdk/jdk-7u75-linux-x64.rpm

rpm -ivhjdk-7u75-linux-x64.rpm

echo "exportJAVA_HOME=/usr/java/jdk1.7.0_75" >>/etc/profile

echo "exportJAVA_BIN=/usr/java/jdk1.7.0_75/bin" >>/etc/profile

echo "exportPATH=$PATH:$JAVA_HOME/bin" >>/etc/profile

echo "exportCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar">>/etc/profile

source /etc/profile

3、安装组件

这里可以选服务，也可以不选服务，进入管理页面在慢慢配置。

三、配置服务

http://192.168.202.34:7180/cmf/home

[root@csmaster ~]# servicecloudera-scm-server status

cloudera-scm-server (pid  1587) is running...

[root@csmaster ~]# servicecloudera-scm-server-db status

pg_ctl: server is running (PID: 1531)

/usr/bin/postgres "-D" "/var/lib/cloudera-scm-server-db/data"

 

1、先安装基础服务zk

2、hdfs

3、hdfs ha配置

4、yarn

5、yarn高可用配置

6、pg集群安装

7、元数据数据库迁移到pg集群。