CDH6.0.1详细安装步骤
系统环境
• 操作系统:3台 CentOS 7.3 x64
• Cloudera Manager:6.0.1
• CDH: 6.0.1
VM Name | VM IP Adress | Centos_HostName | Role |
---|---|---|---|
Centos_wjg1 | 192.168.1.101 | hadoop01 | Cloudera Manager Server + Cloudera Manager Agent |
Centos_wjg2 | 192.168.1.102 | hadoop02 | Cloudera Manager Agent |
Centos_wjg6 | 192.168.1.106 | hadoop06 | Cloudera Manager Agent |
一、配置网络环境
- 编辑/etc/sysconfig/network-scripts/ifcfg-eth0配置文件,修改其中以下红色颜色标示的项目:
vi /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=eth0
UUID=3719cf03-5d05-454b-903f-e1e26ad34fd6
DEVICE=eth0
ONBOOT=yes
IPADDR=192.168.1.101
NETWORK=192.168.1.1
NETSTAT=255.255.255.0
GATEWAY=192.168.1.1
DNS1=222.186.160.70
DNS2=114.114.114.114
2.编辑/etc/default/grub文件,在GRUB_CMD_LINE_LINUX=""项中,插入"net.ifnames=0 biosdevname=0"(与原有项之间用空格隔开),保存退出。
3.调用命令grub2-mkconfig -o /boot/grub2/grub.cfg更新配置,然后直接reboot重启。
这样之后再用ifconfig -a命令查看时,就有eth0网卡存在了。 - 更改hostname
hostnamectl set-hostname hadoop01 - 修改hosts文件
vi /etc/hosts
192.168.1.101 hadoop01
192.168.1.102 hadoop02
192.168.1.106 hadoop06
-
关闭防火墙
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0
systemctl stop firewalld.service
systemctl disable firewalld.service - 配置免密登录:
在每台主机上生成rsa公钥和私钥
ssh-keygen -t rsa
cd .ssh
cat id_rsa.pub >> authorized_keys
登录其他主机,将其他主机的公钥文件内容都拷贝到hadoop01主机上的authorized_keys文件中,命令如下:
[root@hadoop06 .ssh]# ssh-copy-id -i hadoop01
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
wjg@hadoop01's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'hadoop01'"
and check to make sure that only the key(s) you wanted were added.
在每台主机上重复以上步骤,使任意两台主机间都能相互连通。
至此,免密码登录已经设定完成。
- 更改系统时间:
进入home目录,修改.bash_profile
#setup time region
TZ='Asia/Shanghai'
export TZ
或者: tzselect
将当前时间和日期写入BIOS,避免重启后失效
hwclock –w
- 配置NTP同步各台服务器的时间
9.1主节点配置
在配置之前,先使用ntpdate手动同步一下时间,免得本机与对时中心时间差距太大,使得ntpd不能正常同步。这里选用65.55.56.206作为对时中心,ntpdate -u ntp1.aliyun.com
vi /etc/ntp.conf
#Hosts on local network are less restricted.
取消下面一行的注释,并将其中IP和掩码修改为真实环境IP和掩码,此行配置为允许ntp客户端连接的配置
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
找到server 0.centos.pool.ntp.org iburst,并将所有server配置进行注释
添加下面两行内容
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
启动NTP服务
设置后,重启ntpd服务,用ntpstat来检查效果【大概几秒钟就可以看到效果】
#systemctl restart ntpd
#ntpstat
vi /etc/sysconfig/ntpdate
#Set to 'yes' to sync hw clock after successful ntpdate
SYNC_HWCLOCK=yes
9.2 Agent节点配置
修改/etc/ntp.conf
对该文件进行以下内容的修改:
1、注释所有restrict和server配置
2、添加下面注释,需要修改以下的IP为NTP服务器的IP(在本案例中就是server主机的IP)
server 192.168.1.100
将所有主机的ntp服务设置为开机自动启动
chkconfig ntpd on
systemctl enable ntpd.service
二、 配置163 yum源
1)下载repo文件
cd /etc/yum.repos.d/
wget http://mirrors.163.com/.help/CentOS7-Base-163.repo
2)备份并替换系统的repo文件
mv CentOS-Base.repo CentOS-Base.repo.bak
cp CentOS7-Base-163.repo CentOS-Base.repo
3)执行yum源更新命令
yum clean all
yum makecache
yum update
yum install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb –y
yum install httpd mod_ssl openssl-devel python-psycopg2 –y
yum install MySQL-python
yum install perl
yum install autoconf
三、安装MySQL:
查看并卸载CentOS系统自带的MySQL和Mariadb
rpm -qa | grep MySQL
rpm -qa | grep mariadb
rpm -e --nodeps MySQL-python-1.2.5-1.el7.x86_64
rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
下载并上传以下rpm包到主节点
MySQL-client-5.6.42-1.el7.x86_64.rpm
MySQL-devel-5.6.42-1.el7.x86_64.rpm
MySQL-server-5.6.42-1.el7.x86_64.rpm
在主节点上执行以下命令安装MySQL
rpm -ivh MySQL*
安装完成后启动MySQL:
service mysql start
systemctl start mysql.service
修改MySQL的配置文件
vi /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
#Disabling symbolic-links is recommended to prevent assorted security risks;
#to do so, uncomment this line:
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
#InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
获取系统初始密码:
[root@hadoop01 wjg]# cat /root/.mysql_secret
#The random password set for the root user at Tue Dec 18 20:37:59 2018 (local time): pP4xR06U9nPyWrhA
登录MySQL:
[root@hadoop01 wjg]# mysql -uroot -ppP4xR06U9nPyWrhA
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.42
SET PASSWORD FOR 'root'@'localhost' = PASSWORD('xxxxxxx')
grant all on . to root@"%" identified by " xxxxxxx "
设置MySQL开机自启动
systemctl enable mysql.service
初始化MySQL
/usr/bin/mysql_secure_installation
下载mysql-connector-java-5.1.46.tar.gz并解压
在解压目录下找到mysql-connector-java.jar, 复制到所有主机的/usr/share/java目录下
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46.jar /usr/share/java/mysql-connector-java.jar
[root@hadoop01 usr]# ls -l /usr/share/java/mysql-connector-java.jar
-rw-r--r-- 1 root root 1004838 Dec 19 20:45 /usr/share/java/mysql-connector-java.jar
下载Oracle JDK包oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm上传到所有主机并安装
下载地址:https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/
rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
Creating Databases for Cloudera Software
需要建的库有
scm、amon、rman、hue、metastore、sentry、nav、navms、oozie
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm. TO 'scm'@'%' IDENTIFIED BY 'scm@123';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman. TO 'rman'@'%' IDENTIFIED BY 'rman@123';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue. TO 'hue'@'%' IDENTIFIED BY 'hue@123';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore. TO 'metastore'@'%' IDENTIFIED BY 'metastore@123';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry. TO 'sentry'@'%' IDENTIFIED BY 'sentry@123';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav. TO 'nav'@'%' IDENTIFIED BY 'nav@123';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms. TO 'navms'@'%' IDENTIFIED BY 'navms@123';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie. TO 'oozie'@'%' IDENTIFIED BY 'oozie@123';
初始化cloudera manager数据库
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
四、下载并安装Cloudera Manager
https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/
- cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
- cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm
- cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
在主节点hadoop01服务器上安装1,2,3;
在Agent服务器上安装1,3;
在主节点hadoop01上执行:
systemctl start cloudera-scm-server
systemctl restart cloudera-scm-server
查看日志:
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
显示如下表示服务已经正常起来
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
Cloudera Manager将自动下载文件到/opt/cloudera/parcel-repo目录下
CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel
CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha
CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.torrent
manifest.json
在所有Agent节点hadoop01, hadoop02, hadoop06上执行:
配置Agent的Server指向
vi /etc/cloudera-scm-agent/config.ini
[General]
#Hostname of the CM server.
server_host=hadoop01
[Security]
#Use TLS and certificate validation when connecting to the CM server.
use_tls=1
启动cloudera-scm-agent服务:
systemctl status cloudera-scm-agent
systemctl start cloudera-scm-agent
systemctl restart cloudera-scm-agent
Agent Log文件位置:/var/log/cloudera-scm-agent
cloudera-scm-agent.log
tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log
登录Cloudera Manager管理界面
用户名和密码都是admin
Cloudera 安装Parcel提示主机运行状况不良
解决方法
删除agent目录下面的cm_guid文件,并重启失败节点的agent服务。
[root@hadoop06 ~]# cd /var/lib/cloudera-scm-agent/
[root@hadoop06 cloudera-scm-agent]# ll
total 32
-rw------- 1 root root 2 Dec 28 14:27 active_parcels.json
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Dec 28 23:01 agent-cert
-rw-r--r-- 1 root root 36 Dec 28 14:25 cm_guid
-rw------- 1 root root 13813 Dec 29 07:10 response.avro
-rw-r--r-- 1 root root 36 Dec 28 14:25 uuid
[root@hadoop06 cloudera-scm-agent]# rm -rf cm_guid
[root@hadoop06 cloudera-scm-agent]# systemctl restart cloudera-scm-agent
在所有节点上删除cm_guid以后,cloudera manager会自动继续进行安装。
稍微等一段时间后就会看到parcel已经安装到所有主机上并全部激活。
点击继续就会进入检查主机正确性的界面
第1个警告:Cloudera 建议将 /proc/sys/vm/swappiness 设置为最大值 10。当前设置为 30。使用 sysctl 命令在运行时更改该设置并编辑 /etc/sysctl.conf,以在重启后保存该设置。您可以继续进行安装,但 Cloudera Manager 可能会报告您的主机由于交换而运行状况不良。以下主机将受到影响:
查看详细信息
hadoop[01-02, 06]
解决办法:
临时调整: sysctl vm.swappiness=10
永久调整: vi /etc/sysctl.conf
增加下面一行
vm.swappiness=10
第2个警告:已启用透明大页面压缩,可能会导致重大性能问题。请运行“echo never > /sys/kernel/mm/transparent_hugepage/defrag”和“echo never > /sys/kernel/mm/transparent_hugepage/enabled”以禁用此设置,然后将同一命令添加到 /etc/rc.local 等初始化脚本中,以便在系统重启时予以设置。以下主机将受到影响:
查看详细信息
hadoop[01-02, 06]
解决办法:
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
修改/etc/rc.local将上面两行添加进去
vi /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
第3个警告:psycopg2版本太低的处理
安装pip
yum -y install epel-release
yum -y install python-pip
pip install --upgrade pip
yum install postgresql-devel
安装psycopg2
下载psycopg2-2.7.6.1.tar.gz并解压
[root@hadoop01 psycopg2-2.7.6.1]# python setup.py build
[root@hadoop01 psycopg2-2.7.6.1]# python setup.py install
重新运行检查主机,所有项目检查通过
点击继续进入选择安装产品界面
选择所有服务然后点击继续,进入主机角色分配界面
点击按主机查看对主机分配的角色会看得更清楚一些
点击继续进入数据库设置界面,填写先前创建的数据库名称、用户名和密码
点击继续进入审核更改界面
点击继续开始进行安装
安装完成后点击继续,进入Summary界面
至此,Cloudera Manager6.0.1的安装全部顺利完成。下面,让我们赶快登陆进去看一下
转载于:https://blog.51cto.com/5251672/2336981