hadoop集群建设02cdh5.13版本安装

最新推荐文章于 2022-09-09 14:38:04 发布

小的是运维

最新推荐文章于 2022-09-09 14:38:04 发布

阅读量298

点赞数

文章标签： hadoop

本文链接：https://blog.csdn.net/hadoopp/article/details/100033602

版权

1、修改hosts

一批新机器采购上架以后需要安装操作系统，如果用于CDH集群的建设时要注意：新机器主机名称不要出现大写字母如DSJ字母，会在cm页面出现配置告警，与安全认证有关。如果集群启用kerberos时需要注意此事项

[root@whx-1 ~]# cat /etc/hosts

127.0.0.1 localhost

192.168.1.1 whx-1

192.168.1.2 whx-2

192.168.1.3 whx-3

192.168.1.4 whx-4

192.168.1.5 whx-5

192.168.1.6 whx-6

192.168.1.7 whx-7

2、配置免密码登陆

由于集群规模超大所有机器建议免密登陆，待集群配置安装完毕后建议回收取消root免密，免密配置如下：

[root@whx-1 桌面]# ssh-keygen -t rsa -P ""

[root@whx-1 ~]# cd .ssh

[root@whx-1 .ssh]# ls

id_rsa id_rsa.pub

[root@whx-1 .ssh]# ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.1.1

3、关闭防火墙selinux

集群运行时需要关闭所有节点防火墙以及selinux，生产集群不要暴漏在外网中。

[root@whx-1 ~]# systemctl stop firewalld.service && systemctl disable firewalld.service

Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.

可批量关闭，在root家目录创建hostlist文件里面为所有机器ip。

for ip in `cat hostlist`; do ssh $ip "systemctl stop firewalld.service && systemctl disable firewalld.service"; done

关闭selinux

及时生效

[root@whx-1 ~]# setenforce 0

[root@whx-1 ~]# getenforce

Permissive

永久生效

[root@whx-1 ~]# vim /etc/selinux/config

SELINUX=enforcing改成disabled

批量修改

for ip in `cat hostlist` ; do scp /etc/selinux/config root@$ip:/etc/; done

for ip in `cat hostlist`; do ssh $ip "setenforce 0"; done

4、安装jdk

查看机器是否安装jdk

这里建议安装oracle的jdk1.8

首先卸载自带open jdk

[root@whx-1 ~]# rpm -aq |grep jdk

java-1.8.0-openjdk-headless-1.8.0.65-3.b17.el7.x86_64

java-1.8.0-openjdk-1.8.0.65-3.b17.el7.x86_64

批量卸载

for ip in `cat iplist`; do ssh $ip "rpm -e java-1.8.0-openjdk-1.8.0.65-3.b17.el7.x86_64 --nodeps && rpm -e java-1.8.0-openjdk-headless-1.8.0.65-3.b17.el7.x86_64 --nodeps"; done

下载oracle的jdk解压到/usr/java/下

在/etc/profile追加如下，source一下

PATH=$PATH:/usr/java/jdk1.8.0_161

export JAVA_HOME=/usr/java/jdk1.8.0_161

export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH

此处说明下，当集群需要多个版本java时建议不要修改全局变量，可配置用户家目录的.bashrc文件，自定义java路径在/opt/soft/java/下。当用户使用

5、安装mysql数据库

建议使用mysql-mha模式，以下为单节点mysql配置方法。

[root@whx-1 opt]# tar xvf mysql-5.7.18-1.el7.x86_64.rpm-bundle.tar

mysql-community-server-5.7.18-1.el7.x86_64.rpm

mysql-community-embedded-devel-5.7.18-1.el7.x86_64.rpm

mysql-community-devel-5.7.18-1.el7.x86_64.rpm

mysql-community-client-5.7.18-1.el7.x86_64.rpm

mysql-community-common-5.7.18-1.el7.x86_64.rpm

mysql-community-embedded-5.7.18-1.el7.x86_64.rpm

mysql-community-embedded-compat-5.7.18-1.el7.x86_64.rpm

mysql-community-libs-5.7.18-1.el7.x86_64.rpm

mysql-community-server-minimal-5.7.18-1.el7.x86_64.rpm（这个包可以删除）

mysql-community-test-5.7.18-1.el7.x86_64.rpm

mysql-community-minimal-debuginfo-5.7.18-1.el7.x86_64.rpm

mysql-community-libs-compat-5.7.18-1.el7.x86_64.rpm

[root@whx-1 opt]# rm mysql-5.7.18-1.el7.x86_64.rpm-bundle.tar -f

[root@whx-1 opt]# mkdir mysql

[root@whx-1 opt]# mv mysql-community-* mysql

[root@whx-1 opt]# cd mysql/

[root@whx-1 mysql]# createrepo ./

Spawning worker 0 with 12 pkgs

Workers Finished

Saving Primary metadata

Saving file lists metadata

Saving other metadata

Generating sqlite DBs

Sqlite DBs complete

配置mysql的yum源，该步骤主要是方便其他机器安装mysql。

[root@whx-1 mysql]# cd /etc/yum.repos.d/

[root@whx-1 yum.repos.d]# vim mysql.repo

[mysql]

name=5.7

baseurl=file:///opt/mysql/

enable=1

gpgcheck=0

[root@whx-1 yum.repos.d]# yum repolist

源标识源名称状态

mysql 5.7 12

repolist: 12

卸载mariadb的lib

[root@whx-1 yum.repos.d]# rpm -qa | grep mariadb

mariadb-libs-5.5.44-2.el7.centos.x86_64

[root@whx-1 yum.repos.d]# rpm -e mariadb-libs-5.5.44-2.el7.centos.x86_64 –nodeps

安装mysql数据库，此处可注释掉本地center os 7的yum源

[root@whx-1 yum.repos.d]# yum -y install mysql

[root@whx-1 yum.repos.d]# yum -y install mysql-server

破解mysql密码，注意密码复杂程度，文档为安装示例使用弱口令

[root@whx-1 yum.repos.d]# systemctl set-environment MYSQLD_OPTS="--skip-grant-tables"

[root@whx-1 yum.repos.d]# systemctl start mysqld

[root@whx-1 yum.repos.d]# mysql -u root

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 3

Server version: 5.7.18 MySQL Community Server (GPL)

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> flush PRIVILEGES;