黑猴子的家：Cloudera Manager 之环境准备

最新推荐文章于 2024-09-14 19:55:29 发布

黑猴子的家

最新推荐文章于 2024-09-14 19:55:29 发布

阅读量194

点赞数

文章标签：数据库大数据 centos linux mysql

本文链接：https://blog.csdn.net/qq_28652401/article/details/116646389

版权

1、资源分配

将3台虚拟机分配好内存和磁盘

	hadoop102	hadoop103	hadoop104
内存	10GB	2GB	2GB
磁盘	大于50GB	大于50GB	大于50GB

使用root用户搭建Cloudera Manager

2、安装jdk

三台机器都安装

1)卸载现有jdk

(1)查询是否安装java软件

[root@hadoop102 ~]# rpm -qa | grep java
[root@hadoop102 ~]# rpm -qa | grep jdk

(2)卸载jdk

[root@hadoop102 ~]# rpm -e --nodeps java-1.8

语法：rpm –e --nodeps 软件包名字

2)解压jdk

解压jdk到/opt/module目录下

[root@hadoop102 software]$ tar -xzvf jdk-8u144-linux-x64.tar.gz -C /opt/module/

3)配置环境变量

[root@hadoop102 ~]# vim /etc/profile
##JAVA_HOME
export JAVA_HOME=/opt/module/jdk1.8.0_144
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin

[root@hadoop102 ~]# source /etc/profile
[root@hadoop102 ~]# java -version
java version "1.8.0_144"

尖叫提示：环境变量的配置，根据情况在 /etc/profile , /etc/environment, ~/.bashrc 此三个文件中配置,记得分发

3、离线安装MySQL

hadoop102节点安装

1)卸载mysql

(1)关闭mysql服务

[root@hadoop102 ~]# systemctl status mysql
[root@hadoop102 ~]# systemctl stop mysql

(2)查看mysql

[root@hadoop102 ~]# rpm -qa | grep -i mysql
[root@hadoop102 ~]# rpm -qa | grep -i mariadb

(3)卸载mysql

[root@node1 ~]# rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64

2)安装mysql

(1)添加mysql用户

[root@hadoop102 ~]# id mysql
[root@hadoop102 ~]# useradd mysql
[root@hadoop102 ~]# passwd mysql

尖叫提示：这一步一定要做
(2)创建MySQL-5.7.21-rpm文件夹

[root@hadoop102 software]$ mkdir MySQL-5.7.21-rpm

(3)解压mysql的rpm安装包

[root@hadoop102 software]$ tar -xvf MySQL-5.7.21-1.el7.x86_64.rpm-bundle.tar -C ./MySQL-5.7.21-rpm

(4)设置执行权限

[root@hadoop102 software]$ chmod -R 755 ./MySQL-5.7.21-rpm/*

(5)安装mysql rpm包

[root@hadoop102 software]# cd MySQL-5.7.21-rpm
[root@hadoop102 MySQL-5.7.21-rpm]# rpm -ivh mysql-community-common-c-1.el7.x86_64.rpm
[root@hadoop102 MySQL-5.7.21-rpm]# rpm -ivh mysql-community-libs-5.7.21-1.el7.x86_64.rpm
[root@hadoop102 MySQL-5.7.21-rpm]# rpm -ivh mysql-community-client-5.7.21-1.el7.x86_64.rpm
[root@hadoop102 MySQL-5.7.21-rpm]# rpm -ivh mysql-community-server-5.7.21-1.el7.x86_64.rpm
[root hadoop102 MySQL-5.7.21-rpm]# rpm -ivh mysql-community-libs-compat-5.7.21-1.el7.x86_64.rpm

尖叫提示：rpm -ivh --prefix= /opt/moudle MySQL-client-5.7.21-1.el7.x86_64.rpm，rpm安装的时候可以通过--prefix来制定目录安装
(6)修改配置文件

[root@hadoop102 ~]# vim /etc/my.cnf
validate_password_policy=0
validate_password_length=1

(7)启动mysql服务端

[root@hadoop102 ~]# systemctl start mysqld

(8)查看mysql服务端状态

[root@hadoop102 MySQL-5.7.21-rpm]$ systemctl status mysqld

(9)查看mysql的root用户的密码

[root@hadoop102 ~]$ grep 'temporary password' /var/log/mysqld.log
2018-03-15T01:29:11.385674Z 1 [Note] A temporary password is generated for root@localhost: 42mL&hX3G6+a

(10)mysql客户端登录

[root@hadoop102 ~]$ mysql -uroot -p42mL&hX3G6+a

(11)修改密码

mysql> set password=password('000000');
mysql> flush privileges;

(12)退出mysql

mysql> exit;
mysql> quit;

尖叫提示：两种方式都可以

3)MySQL允许远程登录

任何主机上都能登录MySQL数据库
(1)登录mysql客户端

[root@hadoop102 ~]# mysql -uroot -p000000

(2)显示数据库

mysql> show databases;

(3)使用mysql数据库

mysql> use mysql;

(4)展示mysql数据库中的所有表

mysql> show tables;

(5)展示user表的结构

mysql> desc user;

(6)查询user表

mysql> select user,host,authentication_string from user;

(7)修改user表，把Host表内容修改为%

mysql> update user set host='%' where user='root' and host='localhost';
mysql> grant all privileges on *.* to 'root'@'%' identified by '000000' with grant option;

(8)刷新

mysql> flush privileges;

(9)退出

mysql> quit;

(10)设置开机启动

[root@hadoop102 ~]# systemctl enable mysqld

4)创建数据库

mysql> create database hive default charset utf8 collate utf8_general_ci;
mysql> create database amon default charset utf8 collate utf8_general_ci;
mysql> create database hue default charset utf8 collate utf8_general_ci;
mysql> create database monitor default charset utf8 collate utf8_general_ci;
mysql> create database oozie default charset utf8 collate utf8_general_ci;
mysql> flush privileges;

5)修改密码

mysql> set password=password('alex32405MK!@#$%');
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

4、MariaDB-10.1.22_64安装

hadoop102节点安装

1)制作MariaDB.repo

[root@hadoop102 ~]# vim /etc/yum.conf
keepcache=1

[root@hadoop102 ~]# cd /etc/yum.repos.d/
[root@hadoop102 yum.repos.d]# touch MariaDB.repo
[root@hadoop102 yum.repos.d]# vim MariaDB.repo
[MariaDB]
name=MariaDB
baseurl=http://yum.mariadb.org/10.1/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1
enabled=1

2)yum安装MariaDB

[root@hadoop102 ~]# yum install -y MariaDB-server MariaDB-client

3)启动服务

[root@hadoop102 ~]# service mysql start
[root@hadoop102 ~]# systemctl start mariadb

4)开机启动

[root@hadoop102 ~]# systemctl enable mariadb
[root@hadoop102 ~]# chkconfig mysql –list

5)MariaDB设置初始化密码及修改密码

(1)方法1

[root@hadoop102 ~]# mysql
MariaDB[(none)]> set password= password('000000');

尖叫提示：语法MariaDB[(none)]> set password= password('newpassward');
(2)方法2

[root@hadoop102 ~]# mysqladmin -u root password 000000

尖叫提示：语法[root@localhost ~]# mysqladmin -u root password 'newpassword'

如果root已经设置过密码

[root@hadoop102 ~]# mysqladmin -uroot -p000000 password 123456

尖叫提示：语法mysqladmin -uroot -p'oldpassword' password 'newpassword'

6)允许远程访问

MariaDB [(none)]> use mysql;
MariaDB [mysql]> grant all privileges on *.* to 'root'@'%' identified by '000000' with grant option;
MariaDB [mysql]> select host,user,password from user;
MariaDB [mysql]> update user set host='%' where user='root' and host='localhost';
MariaDB [mysql]> delete from user where host!='%';
MariaDB [mysql]> flush privileges;

7)修改Mariadb存储路径

(1)确定MariaDB数据库能正常运行

[root@hadoop102 ~]# service mysql status

(2)确定正常后关闭服务

[root@hadoop102 ~]# service mysql stop

(3)建立要更改数据存放的目录

[root@hadoop102 /]# mkdir /data/mysql_data
[root@hadoop102 /]# chown -R mysql:mysql /data/mysql_data

(4)复制默认数据存放文件夹到/data/mysql_data

[root@hadoop102 /]# cp -a /var/lib/mysql /data/mysql_data

(5)修改server.cnf

[root@hadoop102 /]# vim /etc/my.cnf.d/server.cnf
datadir=/data/mysql_data/mysql
socket=/var/lib/mysql/mysql.sock
#default-character-set=utf8
character_set_server=utf8
slow_query_log=on
slow_query_log_file=/data/mysql_data/slow_query_log.log
long_query_time=2

尖叫提示：退出保存Esc ->：wq如果忘记关掉selinux了，导致更换路径后，无法启动mysql"datadir"、"innodb_data_home_dir"、"innodb_log_group_home_dir"都修改到最新的"mysql"数据库目录地址

8)添加mysql的Jdbc驱动

[root@hadoop102 /]#cp mysql-connector-java-5.1.27-bin.jar /usr/share/java/mysql-connector-java.jar

分发

[root@hadoop102 java]# xsync mysql-connector-java.jar

9)创建数据库

MariaDB [mysql]> create database hive default charset utf8 collate utf8_general_ci;
MariaDB [mysql]> create database amon default charset utf8 collate utf8_general_ci;
MariaDB [mysql]> create database hue default charset utf8 collate utf8_general_ci;
MariaDB [mysql]> create database monitor default charset utf8 collate utf8_general_ci;
MariaDB [mysql]> create database oozie default charset utf8 collate utf8_general_ci;
MariaDB [mysql]> flush privileges;

5、Yum安装MySQL

1)获取rpm包

[root@hadoop104 ~]# cd /etc/yum.repos.d/
[root@hadoop102 yum.repos.d]$ wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm

2)安装mysql.rpm包

[root@hadoop102 yum.repos.d]$ rpm -ivh mysql-community-release-el7-5.noarch.rpm

尖叫提示：安装rpm文件，获得两个repo文件(master+slave)mysql-community.repomysql-community-source.repo

3)安装MySQL

[root@hadoop102 yum.repos.d]$ yum install -y mysql-server

4)查看mysql都安装了什么服务

[root@hadoop102 ~]$ rpm -qa | grep -i mysql
mysql-community-client-5.6.49-2.el7.x86_64
mysql-community-common-5.6.49-2.el7.x86_64
mysql-community-release-el7-5.noarch
mysql-community-libs-5.6.49-2.el7.x86_64
mysql-community-server-5.6.49-2.el7.x86_64

5)启动mysql服务

[root@hadoop102 ~]$ systemctl start mysqld.service
[root@hadoop102 ~]$ systemctl enable mysqld.service
[root@hadoop102 ~]$ systemctl status mysqld.service

6)登陆mysql

[root@hadoop102 ~]$ mysql -uroot -p
Enter password: (没有密码)
mysql> set password = password('123456');
mysql> flush privileges;
mysql> exit;

7)远程登陆

mysql> show databases;
mysql> use mysql;
mysql> show tables;
mysql> desc user;
mysql> select host,user,password from user;
mysql> update user set host = '%' where user = 'root' and host = 'localhost';

grant all privileges on *.* to 'root'@'%' identified by '123456';

6、配置SSH免密登陆

1)生成公钥私钥

[root@hadoop102 ~]# ssh-keygen -t rsa
[root@hadoop103 ~]# ssh-keygen -t rsa
[root@hadoop104 ~]# ssh-keygen -t rsa

[root@hadoop102 ~]# ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa
[root@hadoop103 ~]# ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa
[root@hadoop104 ~]# ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa

2)auto-ssh-expect.sh

[root@hadoop102 ~]# yum install -y sshpass expect
[root@hadoop103 ~]# yum install -y sshpass expect
[root@hadoop104 ~]# yum install -y sshpass expect

[root@hadoop102 ~]# cd /usr/local/bin/
[root@hadoop102 bin]# touch auto-ssh-expect.sh
[root@hadoop102 bin]# chmod 755 auto-ssh-expect.sh
[root@hadoop102 bin]# vim auto-ssh-expect.sh
#!/bin/bash
user=`whoami`
passwd=000000
#yum install -y sshpass expect
for((current=102; current<=104; current++));do
  for((host=102; host<=104; host++));do
    sshpass -p $passwd ssh -q -o StrictHostKeyChecking=no $user@hadoop$current expect << EOF
    spawn ssh-copy-id ${user}@hadoop${host}
    expect {
      "yes/no" {send "yes\r";exp_continue}
     "password" {send "$passwd\r"}
    }
    expect eof
EOF
  done
done
[root@hadoop102 ~]# auto-ssh-expect.sh

3)auto-ssh-sshpass.sh

[root@hadoop102 ~]# yum install -y sshpass
[root@hadoop103 ~]# yum install -y sshpass
[root@hadoop104 ~]# yum install -y sshpass

[root@hadoop102 ~]# cd /usr/local/bin/
[root@hadoop102 bin]# touch auto-ssh-sshpass.sh
[root@hadoop102 bin]# chmod 755 auto-ssh-sshpass.sh
[root@hadoop102 bin]# vim auto-ssh-sshpass.sh
#!/bin/bash
user=`whoami`
passwd=000000
#yum install -y sshpass
echo "开始配置免密登录......"
for((current=102; current<=104; current++));do
  for((host=102; host<=104; host++));do
    sshpass -p $passwd ssh -q -o StrictHostKeyChecking=no $user@hadoop$current "sshpass -p $passwd ssh-copy-id -o StrictHostKeyChecking=no $user@hadoop$host"
  done
done
echo "恭喜, 免密登录配置完成!"
[root@hadoop102 ~]# auto-ssh-sshpass.sh

7、NTP时间同步

1)时间服务器配置(必须root用户)

(1)检查ntp是否安装

[root@hadoop102 ~]# rpm -qa | grep ntp
fontpackages-filesystem-1.44-8.el7.noarch
ntp-4.2.6p5-25.el7.centos.2.x86_64
python-ntplib-0.3.2-1.el7.noarch
ntpdate-4.2.6p5-25.el7.centos.2.x86_64

(2)如果没有ntp服务，可使用yum命令进行安装

[root@hadoop102 ~]# yum -y install ntp

2)检查当前系统时区

(1)选择某台机器，作为集群中时间服务器的主节点，然后其他机器同步该机器的时间即可。但是在开始这步操作之前，我们需要确保所有节点的时区是统一的

[root@hadoop102 ~]# date -R

显示类似如下格式：
Wed, 28 Feb 2018 15:28:53 +0800

(2)时区
尖叫提示：如果显示的时区不是+0800，你可以删除localtime文件夹后，再关联一个正确时区的链接过去
``` scala
[root@hadoop102 ~]# rm -rf /etc/localtime
[root@hadoop102 ~]# ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

3)同步网络时间

如果怀疑自己本地机器的时间与标准时间相差很多，建议使用时间服务器的主节点同步一下网络时间：

[root@hadoop102 ~]# ntpdate pool.ntp.org

4)修改ntp配置文件

(1)我们需要修改ntp服务的配置文件，关闭网络时间的同步

[root@hadoop102 ~]# vi /etc/ntp.conf

(2)对如下内容做出修改

# 授权192.168.2.0网段上的所有机器可以从这台机器上查询和同步时间
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap

# 当该节点丢失网络连接，依然可以作为时间服务器为集群中的其它节点提供时间同步
server 127.127.1.0
fudge 127.127.1.0 stratum 10

#集群在局域网中，不使用其他的网络时间
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

尖叫提示：nomodify：客户端不能使用ntpc与ntpq修改服务器的时间参数notrap：不提供trap远程时间登录的功能

5)重启ntp服务

(1)CentOS6

[root@hadoop102 ~]# service ntpd restart
[root@hadoop102 ~]# chkconfig ntpd on

(2)CentOS7

[root@hadoop102 ~]# systemctl restart ntpd.service
[root@hadoop102 ~]# systemctl enable ntpd.service

6)设置定时同步任务

(1)首先在其它节点上关闭ntp服务
CentOS6

[root@hadoop103 ~]# service ntpd stop
[root@hadoop103 ~]# chkconfig ntpd off

CentOS7

[root@hadoop103 ~]# systemctl stop ntpd.service
[root@hadoop103 ~]# systemctl disable ntpd.service

查看ntp进程id

[root@hadoop103 ~]# pgrep ntpd

(2)其它节点手动同步第一台时间服务器的时间进行测试

[root@hadoop103 ~]# ntpdate hadoop102

(3)其它节点制定计划任务，周期性同步时间

[root@hadoop103 ~]# crontab -e
# .------------------------------------------minute(0~59)
# | .----------------------------------------hours(0~23)
# | | .--------------------------------------day of month(1~31)
# | | | .------------------------------------month(1~12)
# | | | | .----------------------------------day of week(0~6)
# | | | | | .--------------------------------command
# | | | | | |
# | | | | | |
0 0 * * * /usr/sbin/ntpdate hadoop102

(4)重启定时任务
CentOS6

[root@hadoop102 ~]# service crond restart