大数据部署手册(个人)
安装包下载
MySQL 下载:
下载mysql地址:http://dev.mysql.com/downloads/mysql/5.6.html#downloads
CDH 下载
下载cdh-5.4.3放置到/usr/local/soft/下
http://archive.cloudera.com/cdh5/parcels/5.4.3/
CDH-5.4.3-1.cdh5.4.3.p0.6-el6.parcel
CDH-5.4.3-1.cdh5.4.3.p0.6-el6.parcel.sha
manifest.json("parcelName": " CDH-5.4.3-1.cdh5.4.3.p0.6-el6.parcel ",要与包名一致)
kafka 下载
http://archive.cloudera.com/kafka/parcels/3/
服务器
hosts 配置
vim /etc/hosts
192.168.1.201 hx1
192.168.1.202 hx2
192.168.1.203 hx3
192.168.1.204 hx4
192.168.1.205 hx5
关闭防火墙
vim /etc/selinux/config
#SELINUX=enforcing (注释掉)
SELINUX=disabled (换成disabled)
免密配置
ssh-keygen -t rsa(三次回车)
cd /root/.ssh/
不存在就创建 authorized_keys
touch authorized_keys 创建空文件
然后将每个节点的/root/.ssh/id_rsa.pub 内容复制到其中一个节点的authorized_keys里面
发送authorized_keys到其他节点
scp authorized_keys 发送到/root/.ssh/
先通过一台服务登录其他的服务器,然后会自动生成一个为 /root/.ssh/known_hosts 文件,然后将known_hosts 文件scp 到其他的服务器上。(这样就不用逐个的去确认免密)
时间同步
集群时间同步:同步时间每天的00:01,同步日志存放在/opt/systime.log
Vim systime.sh
#!/bin/sh
Cur=’date +%H:%M:%S’
Echo “sync time starting : $Cur”
for i in {1..5}; do echo s1-hx${i}; ssh hx${i} “source /opt/profile; date –s $ Cur”;done;
Echo “sync time starting : $Cur”
在/etc/目录下使用:crontab –e
1 0 * * * /etc/systime.sh >>/opt/systime.log (1 0 * * *中间有空格)
后期可以配置为NTP同步
https://www.linuxidc.com/Linux/2015-11/124911.htm
安装 MySQL
删除过期文件
1.rpm -qa|grep -i mysql 查看安装的mysql
2./usr/local/mysql/support-files/mysql.server stop 停止mysql服务
3.whereis mysql 收集mysql文件夹信息 或者 find / -name mysql(推荐)
4.rpm -ev mysql-***** 卸载删除mysql组件
5.再 rm -rf 删除mysql对应文件夹
6.删除mysql用户名和用户组
more /etc/passwd | grep mysql
more /etc/shadow | grep mysql
more /etc/group | grep mysql
userdel mysql
groupdel mysql7.rpm -qa | grep -i mysql再确认是否删除8.配置文件一般有 /etc/my.cnf /var/log/mysqld.log /var/run/mysqld/ 根据具体情况删除
部署 MySQL
下载mysql文件压缩包 放到、/usr/local/
cd /usr/local/
tar -zxvf mysql**** 解压
mv mysql**** mysql 重命名
groupadd mysql 创建用户组
useradd -r -g mysql mysql 创建用户添加进用户组 -r标识系统内部用户
chown -R mysql mysql 给用户加mysql文件权限
chgrp -R mysql mysql 给用户组mysql文件权限
yum install libaio 必须先安装这个
touch /var/log/mysqld.log 添加日志文件
chmod 777 /var/log/mysqld.log 再赋权
chown mysql:mysql /var/log/mysqld.log
mkdir /var/run/mysqld
touch /var/run/mysqld/mysqld.pid
chmod 777 /var/run/mysqld/mysqld.pid
chown mysql:mysql /var/run/mysqld/mysqld.pid
chmod 777 -R /var/run/mysqld
- 创建配置文件 添加配置 vim /etc/my.cnf
[client]
port = 3306
socket = /tmp/mysql.sock
[mysqld]
character_set_server=utf8
init_connect='SET NAMES utf8'
basedir=/usr/local/mysql
datadir=/usr/local/mysql/data
socket=/tmp/mysql.sock
#skip-grant-tables
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
lower_case_table_names = 1
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
max_connections=5000
default-time_zone = '+8:00'
初始化数据库
/usr/local/mysql/bin/mysqld --initialize --user=mysql --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --lc_messages_dir=/usr/local/mysql/share --lc_messages=en_US
- 查看初始密码
cat /var/log/mysqld.log
- 启动MySQL服务
/usr/local/mysql/support-files/mysql.server start
- 修改初始密码
/usr/local/mysql/bin/mysql -uroot -p初始密码
- 远程访问授权
grant all privileges on *.* to '用户名'@'%' identified by '登录密码' with grant option;
flush privileges;
创建 cm 组件需要的 数据库
创建集群监控数据库
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
创建hive数据库
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
创建hue数据库
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
安装cloudera-manager-5.4.3
ll /usr/local/soft/
cloudera-manager-el6-cm5.4.3_x86_64.tar.gz
ll /usr/local/soft/
CDH-5.4.3-1.cdh5.4.3.p0.6-el6.parcel
CDH-5.4.3-1.cdh5.4.3.p0.6-el6.parcel.sha
manifest.json
重命名:(mv CDH-5.4.3-1.cdh5.4.3.p0.6-el6.parcel.sha1 CDH-5.4.3-1.cdh5.4.3.p0.6-el6.parcel.sha)
解压cm-5.10.2到/opt下
tar -zxvf /usr/local/soft/cloudera-manager-el6-cm5.4.3_x86_64.tar.gz -C /opt/
创建scm用户(在每个节点上都要)
useradd --system --home=/opt/cm-5.13.2/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
vim /opt/cm-5.13.2/etc/cloudera-scm-agent/config.ini
设置server_host=s1 (每个节点设置为主节点别名或者ip)
cp /usr/local/soft/mysql-connector-java-5.1.34-bin.jar /opt/cm-5.13.2/share/cmf/lib/
创建cm数据库(只在主节点创建)
/opt/cm-5.13.2/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -pHlhy@2017 --scm-host localhost scm root Hlhy@2017
复制到s2~s6节点
for i in {2..6}; do echo hx${i}; scp -r /opt/c* root@hx${i}:/opt/; done; (中途不可停止,否则删除已拷贝节点文件重新执行)
复制CDH到/opt/cloudera/parcel-repo/下(只在s1上复制,其他节点不需要)
CDH-5.4.3-1.cdh5.4.3.p0.6-el6.parcel
CDH-5.4.3-1.cdh5.4.3.p0.6-el6.parcel.sha
manifest.json
二 、
启动cm server(s1上)
/opt/cm-5.13.2/etc/init.d/cloudera-scm-server start(start|stop|status)
启动cm agent(s1~s6上)
/opt/cm-5.13.2/etc/init.d/cloudera-scm-agent start(start|stop|status)
使用s1:7180中开始配置集群,如下:
登录CDH平台账号admin密码admin