cdh6安装

最新推荐文章于 2024-05-14 08:40:42 发布

chixuegui9567

最新推荐文章于 2024-05-14 08:40:42 发布

阅读量1.6k

点赞数 1

文章标签：数据库开发工具 java

原文链接：https://my.oschina.net/whx403/blog/2999820

版权

打通SSH，设置ssh无密码登陆（所有节点）

1、在每台服务器上都执行ssh-keygen -t rsa生成密钥对:				
ssh-keygen -t rsa，一路回车，生成无密码的密钥对
//将公钥添加到认证文件中：cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
//设置authorized_keys的访问权限：chmod 600 ~/.ssh/authorized_keys
2、在每台服务器上生成密钥对后，将公钥复制到其它结点上：
ssh-copy-id -i  ~/.ssh/id_rsa.pub root@hd241
ssh-copy-id -i  ~/.ssh/id_rsa.pub root@hd242

配置主机名和host（所有节点）

1、vim /etc/hostname -> sjhl-ad-hd1
2、hostnamectl set-hostname sjhl-ad-hd1 # 使其立刻生效
3、vim /etc/hosts 增加
192.168.1.29    sjhl-ad-hd1
192.168.1.30    sjhl-ad-hd2
192.168.1.185   sjhl-ad-hd3
192.168.1.186   sjhl-ad-hd4
192.168.1.187   sjhl-ad-hd5
192.168.1.188   sjhl-ad-hd6
192.168.1.189   sjhl-ad-hd7
192.168.1.190   sjhl-ad-hd8
(
4、配置 vim /etc/sysconfig/network
NETWORKING_IPV6=no
PEERNTP=no
HOSTNAME=sjhl-ad-hd1
重启网络：service network restart
测试：host -v -t A $(hostname)
)

~~在线安装-配置 Cloudera Manager 仓库(所有节点)~~ wget https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/ rpm --import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera

离线安装

1、CM6 RPM：https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/ 
下载
├── cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
├── cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
├── cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm
└── cloudera-manager-server-db-2-6.0.1-610811.el7.x86_64.rpm

2、下载ASC文件：https://archive.cloudera.com/cm6/6.0.1/allkeys.asc

3、CDH6 Parcels：https://archive.cloudera.com/cdh6/6.0.1/parcels/
下载CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel和manifest.json这两个文件

配置Cloudera Manager yum库（一台master节点）

1、安装httpd和createrepo：
yum -y install httpd createrepo

2、启动httpd服务并设置开机自启动：
systemctl start httpd
systemctl enable httpd

3、在/var/www/html/下创建目录
mkdir /var/www/html/cloudera-repos/

4、把下载的CM6 RPM 和 ASC文件 移到 /var/www/html/cloudera-repos/
/var/www/html/cloudera-repos/
├── allkeys.asc
├── cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
├── cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
├── cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm
├── cloudera-manager-server-db-2-6.0.1-610811.el7.x86_64.rpm

5、cd /var/www/html/cloudera-repos/
生成RPM元数据：
createrepo .

[root@sjhl-ad-hd1 cloudera-repos]# createrepo .
Spawning worker 0 with 2 pkgs
Spawning worker 1 with 2 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

6、确保通过浏览器http://sjhl-ad-hd1/cloudera-repos/ 可以看到
或者：curl http://sjhl-ad-hd1/cloudera-repos/
<html>
<head>
  <title>Index of /cloudera-repos</title>
 </head>
 <body>
<h1>Index of /cloudera-repos</h1>
  <table>
   <tr><th valign="top"><img src="/icons/blank.gif" alt="[ICO]"></th><th><a href="?C=N;O=D">Name</a></th><th><a href="?C=M;O=A">Last modified</a></th><th><a href="?C=S;O=A">Size</a></th><th><a href="?C=D;O=A">Description</a></th></tr>
   <tr><th colspan="5"><hr></th></tr>
<tr><td valign="top"><img src="/icons/back.gif" alt="[PARENTDIR]"></td><td><a href="/">Parent Directory</a>       </td><td>&nbsp;</td><td align="right">  - </td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/text.gif" alt="[TXT]"></td><td><a href="allkeys.asc">allkeys.asc</a>            </td><td align="right">2018-10-11 20:35  </td><td align="right"> 14K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm">cloudera-manager-age..&gt;</a></td><td align="right">2018-12-19 18:22  </td><td align="right"> 25M</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm">cloudera-manager-dae..&gt;</a></td><td align="right">2018-12-19 18:22  </td><td align="right">1.0G</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm">cloudera-manager-ser..&gt;</a></td><td align="right">2018-12-19 18:24  </td><td align="right">8.0K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="cloudera-manager-server-db-2-6.0.1-610811.el7.x86_64.rpm">cloudera-manager-ser..&gt;</a></td><td align="right">2018-12-19 18:24  </td><td align="right"> 11K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/folder.gif" alt="[DIR]"></td><td><a href="repodata/">repodata/</a>              </td><td align="right">2018-12-20 09:29  </td><td align="right">  - </td><td>&nbsp;</td></tr>
   <tr><th colspan="5"><hr></th></tr>
</table>
</body>
</html>

配置本地源（所有节点）

1、vim /etc/yum.repos.d/cloudera-manager.repo
编辑以下内容：
[cloudera-manager]
name = Cloudera Manager, Version
baseurl = http://sjhl-ad-hd1/cloudera-repos/
gpgcheck = 1

2、保存，退出,然后执行yum clean all && yum makecache命令

关闭防火墙或者设置iptables集群内部白名单（所有节点）

service iptables stop

systemctl stop firewalld.service
systemctl disable firewalld.service

查看iptables规则：iptables -L -n -v --line-number

关闭SELinux（所有节点）

sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
setenforce 0

禁用透明大页面压缩（所有节点）

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

禁用交换分区（所有节点）

echo "vm.swappiness = 0" >> /etc/sysctl.conf
sysctl -p

修改文件句柄数和缓存页（所有节点）

vim /etc/security/limits.conf

*       soft    nofile  655350
*       hard    nofile  655350
*       soft    nproc   102400
*       hard    nproc   102400
*       soft    memlock unlimited
*       hard    memlock unlimited

安装jdk8（所有节点）

1、下载地址：https://www.oracle.com/technetwork/java/javase/downloads/java-archive-javase8-2177648.html

2、解压，如果不是安装在/usr/java/ 下，创建软连接
ln -s /data/jdk1.8.0_181 /usr/java/jdk1.8.0_181

2、设置环境变量
JAVA_HOME=/usr/java/jdk1.8.0_181
JRE_HOME=$JAVA_HOME/jre
CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export JAVA_HOME JRE_HOME CLASS_PATH PATH

下载mysql jdbc 驱动（所有节点）

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar zxvf mysql-connector-java-5.1.46.tar.gz

mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

安装ntp服务（所有节点）

yum install ntp
systemctl start ntpd  
chkconfig ntpd on #设置开机启动

1、选master作为ntp服务，vim /etc/ntp.conf 配置如下：

driftfile /var/lib/ntp/drift   #这是预设时间差异分析文件，我们暂时用不到，就这样留着
pidfile    /var/run/ntpd.pid
logfile    /var/log/ntp.log


restrict    default kod nomodify notrap nopeer noquery   #拒绝ipv4的用户
restrict -6 default kod nomodify notrap nopeer noquery   #拒绝ipv6的用户
restrict 127.0.0.1  #放行本机来源
restrict 192.168.0.0 mask 255.255.0.0 nomodify notrap nopeer noquery
restrict 120.133.0.0 mask 255.255.255.0 nomodify notrap nopeer noquery
# restrict 0.0.0.0 mask 0.0.0.0  nomodify   放行所有来源，这里可以写192.169.0.0. mask 255.255.255.0 只放行一个网断


server 127.127.1.0  #外界同步源无法联系时，使用本地时间为同步服务
fudge  127.127.1.0 stratum 10  #这行是时间服务器的层次。设为0则为顶级，如果要向别的NTP服务器更新时间，请不要把它设为0

server 0.cn.pool.ntp.org prefer  #以这个主机最为优先
server ntp1.aliyun.com iburst minpoll 4 maxpoll 10
server ntp2.aliyun.com iburst minpoll 4 maxpoll 10
server ntp3.aliyun.com iburst minpoll 4 maxpoll 10
server ntp4.aliyun.com iburst minpoll 4 maxpoll 10
server ntp5.aliyun.com iburst minpoll 4 maxpoll 10
server ntp6.aliyun.com iburst minpoll 4 maxpoll 10

2、其他节点配置为客户端即可
driftfile /var/lib/ntp/drift
pidfile    /var/run/ntpd.pid
logfile    /var/log/ntp.log

restrict    default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict 192.168.0.0 mask 255.255.0.0 nomodify notrap nopeer noquery
restrict 120.133.0.0 mask 255.255.255.0 nomodify notrap nopeer noquery


server 127.127.1.0
fudge  127.127.1.0 stratum 10

server sjhl-ad-hd1 prefer
server sjhl-ad-hd2 iburst
server 0.cn.pool.ntp.org iburst minpoll 4 maxpoll 10

3、为每个节点增加手动配置 crontab -e
01 * * * * /usr/sbin/ntpdate 0.cn.pool.ntp.org >/tmp/updateclock.log 2>&1

4、查看同步状态： ntpdc -np

安装mysql（CM Server节点）

1、CentOS 7.6 自带的 MariaDB：
yum install mariadb-server -y

修改mysql配置（vim /etc/my.cnf）

[mysqld]
datadir=/data/mysql/data
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED

symbolic-links=0

key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550

log_bin=/data/mysql/log_bin

server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M


# InnoDB settings
# innodb_file_per_table = 1
# innodb_flush_log_at_trx_commit  = 2
# innodb_log_buffer_size = 64M
# innodb_buffer_pool_size = 4G
# innodb_thread_concurrency = 8
# innodb_flush_method = O_DIRECT
# innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

!includedir /etc/my.cnf.d

启动数据库

chown -R mysql:mysql /data/mysql/

systemctl start mariadb
systemctl enable mariadb

初始化数据库

/usr/bin/mysql_secure_installation

按照下面提示输入

[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] Y
New password:
Re-enter new password:
[...]
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
[...]
All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

创建数据库

#初始化密码
mysqladmin -u root password "pwd"
#set password = password('pwd');
mysql -u root –p

grant all privileges on *.* to 'root'@'%' identified by 'pwd';
flush privileges;

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'pwd';

FLUSH PRIVILEGES;

安装CM Server（server 节点）

yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

安装CM agent（所有节点）

#先安装，后面通过cm安装的时候会快，不用等待
yum install cloudera-manager-daemons cloudera-manager-agent

配置本地Parcel（CM Server 节点）

1、cm server安装后，进入Parcel目录
cd /opt/cloudera/parcel-repo

2、将之前下载的Parcel文件移进该目录
CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel
manifest.json

3、执行命令生成sha文件
sha1sum CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel | awk '{ print $1 }' > CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha

4、修改文件权限
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*

/opt/cloudera/parcel-repo
├── CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel
├── CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha
├── manifest.json

设置Cloudera Manager 数据库（CM Server 节点）

执行命令：/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

启动Cloudera Manager Server服务

systemctl start cloudera-scm-server

#监控服务启动状态
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

访问Cloudera Manager WEB界面

http://sjhl-ad-hd1:7180/cmf/login

转载于:https://my.oschina.net/whx403/blog/2999820

chixuegui9567

关注

1
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
cdh6安装

打通SSH，设置ssh无密码登陆（所有节点）1、在每台服务器上都执行ssh-keygen -t rsa生成密钥对: ssh-keygen -t rsa，一路回车，生成无密码的密钥对//将公钥添加到认证文件中：cat ~/.ssh/id_rsa.pub >> ~/.ssh...
复制链接

扫一扫