用cm搭建Hadoop环境

 

1、下载

1、CM6.1安装包

https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPM-GPG-KEY-cloudera

https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm

https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm

https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm

https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-db-2-6.1.0-769885.el7.x86_64.rpm

 

https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

放到master的/var/www/html/cloudera-repos/cm6/6.1.0

2、CDH6.1的parcel安装包

https://archive.cloudera.com/cdh6/6.1.0/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel

https://archive.cloudera.com/cdh6/6.1.0/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha256

https://archive.cloudera.com/cdh6/6.1.0/parcels/manifest.json

放到master的/opt/cloudera/parcel-repo/

3、文档

https://www.cloudera.com/documentation/enterprise/latest/topics/cm_intro_primer.html

2、环境

Master节点内存>=8G。

host	mysql	cm	cdh
192.168.1.21	*		master
192.168.1.22		cm service	second
192.168.1.23			worker
192.168.1.24			worker
192.168.1.25			worker

3、准备

3.1、配置hostname

配置域名相关的信息。

1、每个节点的hostname，等效修改/etc/hostname文件

hostnamectl set-hostname  cm01

hostnamectl set-hostname  cm02

hostnamectl set-hostname  cm03

hostnamectl set-hostname  cm04

hostnamectl set-hostname  cm05

2、每个节点的/etc/hosts

cat /etc/hosts

192.168.1.21   cm01

192.168.1.22   cm02

192.168.1.23   cm03

192.168.1.24   cm04

192.168.1.25   cm05

3、如果用域名，修改/etc/sysconfig/hostname

hostname=domain-name

uname –a需要和hostname得到一致的域名。

 

3.2、禁用防火墙

systemctl disable firewalld

 

3.3、设置SELinux

vi /etc/sysconfig/selinux

SELINUX=disabled

重启生效。

 

3.4、启用NTP

配置NTP服务(实际未用)，虚拟机可在选项中与主机同步。

1、安装NTP

yum -y install ntp

2、配置NTP

master配置，(阿里)time1.aliyun.com

vi /etc/ntp.conf

server ntp.sjtu.edu.cn prefer(复旦大学ntp)

slave配置

server master(同步master)

3、开启

systemctl start ntpd

3、查看同步效果

ntpstat

 

3.5、禁用透明大页面压缩

vi /etc/rc.local

echo never > /sys/kernel/mm/transparent_hugepage/defrag

echo never > /sys/kernel/mm/transparent_hugepage/enabled

 

/etc/rc.local是/etc/rc.d/rc.local的软连接。

chmod 777 /etc/rc.d/rc.local

 

3.6、优化交换分区

vi /etc/sysctl.conf

vm.swappiness = 10

vm.max_map_count=262144

 

执行生效

sysctl -p /etc/sysctl.conf

 

3.7、SSH免密

ssh-keygen -t rsa

 

chmod 711 .ssh

cat id_rsa.pub >> authorized_keys

chmod 644 authorized_keys

 

ssh-copy-id id_rsa.pub root@cm02

ssh-copy-id id_rsa.pub root@cm03

ssh-copy-id id_rsa.pub root@cm04

ssh-copy-id id_rsa.pub root@cm05

所有主机互操作一遍。

 

3.8、配置JDK

所有主机rpm安装cloudera的jdk(已下载)。

vi /etc/profile

export JAVA_HOME="/usr/java/jdk1.8.0_141-cloudera"

export CLASSPATH=".:${JAVA_HOME}/lib:${CLASSPATH}"

export PATH="${JAVA_HOME}/bin:${PATH}"

 

. /etc/profile

4、mysql

 

4.1、rpm

mysql的安装包http://repo.mysql.com/yum

yum -y install net-tools perl

与mariadb-libs冲突，卸载rpm -e --nodeps

安装common，libs，client，compat。

安装server。

默认目录

客户端程序和脚本：/usr/bin

mysqld服务器：/usr/sbin

数据：/var/lib/mysql/

错误消息：/usr/share/mysql

配置文件：/etc/my.cnf

 

4.2、配置

1、编辑/etc/my.cnf，官推配置

[mysqld]

server_id=100

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

transaction-isolation = READ-COMMITTED

symbolic-links = 0

key_buffer_size = 32M

max_allowed_packet = 32M

thread_stack = 256K

thread_cache_size = 64

query_cache_limit = 8M

query_cache_size = 64M

query_cache_type = 1

max_connections = 550

log_bin=/var/lib/mysql/mysql_binary_log

binlog_format = mixed

read_buffer_size = 2M

read_rnd_buffer_size = 16M

sort_buffer_size = 8M

join_buffer_size = 8M

# InnoDB settings

innodb_file_per_table = 1

innodb_flush_log_at_trx_commit = 2

innodb_log_buffer_size = 64M

innodb_buffer_pool_size = 4G

innodb_thread_concurrency = 8

innodb_flush_method = O_DIRECT

innodb_log_file_size = 512M

 

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

 

[mysqld_safe]

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

2、检查依赖(自带)

rpm -qa | grep libaio

yum -y install libaio

 

4.3、启停

1、启动

systemctl start mysqld

2、查看临时密码

grep 'temporary password' /var/log/mysqld.log

3、修改密码

mysql -uroot -p

#更改密码策略

mysql> set global validate_password_policy=0;

mysql> set global validate_password_length=1;

#更改密码

mysql> alter user root@localhost identified by 'root';

#赋权远程访问

mysql> grant all privileges on *.* to root@'%' identified by 'root';

mysql> flush privileges;

4、丢失root密码

4.1、在my.cnf的[mysqld]中加入skip-grant-tables=1。

4.2、systemctl start mysqld

4.3、mysql -uroot -p，空密码进入，use mysql。

4.4、update user set authentication_string = password('root'), password_expired='N', password_last_changed=now() where user='root';

4.5、去掉skip-grant-tables。

4.6、另一种方法，建一个文件(未试)。

vi change_password.sql，内容。

alter user