Kylin4.0 beta版部署

Kylin4.0 beta版部署

新特性

• Spark唯一构建引擎
• 引入parquet, 正在踢出HBase
• 可存储到HDFS

1. 准备机器

• 准备三台机器, 系统CentOS7, 切记, 后续安装CDH版本为6.3.2, 高版本系统不支持
• 改hosts, ssh, 免密登录
• 改用户可打开文件数量

* vi /etc/security/limits.conf
*    soft    nofile   32728
*    hard    nofile   1024999
*    soft    nproc    65535
*    hard    noroc    unlimited
*    soft    memlock    unlimited
*    hard    memlock    unlimited
* sysctl -p

• 禁用透明大页面压缩

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

并将上面的两条命令写入开机自启动/etc/rc.local。

• 设置swap空间(所有节点)

echo "vm.swappiness = 0" >> /etc/sysctl.conf

2. 准备数据库

• 主节点安装Mysql
• 创建用户,数据库, 后续使用, 安装什么服务就建对应的数据库, 或者到后续配置前创建

-- cloudera-manager
CREATE DATABASE scm DEFAULT CHARACTER SET utf8;
CREATE USER 'scm'@'%'IDENTIFIED BY 'Kylin@2021!';
GRANT ALL PRIVILEGES ON scm.* TO 'scm'@'%' IDENTIFIED BY 'Kylin@2021!'; 
GRANT ALL PRIVILEGES ON *.* TO 'scm'@'%' IDENTIFIED BY 'Kylin@2021!' WITH GRANT OPTION;
-- active_monitor
CREATE DATABASE active_monitor DEFAULT CHARACTER SET utf8;
CREATE USER 'active_monitor'@'%'IDENTIFIED BY 'Kylin@2021!';
GRANT ALL PRIVILEGES ON active_monitor.* TO 'active_monitor'@'%' IDENTIFIED BY 'Kylin@2021!'; 
GRANT ALL PRIVILEGES ON *.* TO 'active_monitor'@'%' IDENTIFIED BY 'Kylin@2021!' WITH GRANT OPTION;
-- amon
CREATE DATABASE amon DEFAULT CHARACTER SET utf8;
CREATE USER 'amon'@'%'IDENTIFIED BY 'Kylin@2021!';
GRANT ALL PRIVILEGES ON amon.* TO 'amon'@'%' IDENTIFIED BY 'Kylin@2021!'; 
GRANT ALL PRIVILEGES ON *.* TO 'amon'@'%' IDENTIFIED BY 'Kylin@2021!' WITH GRANT OPTION;
-- hive
CREATE DATABASE hive DEFAULT CHARACTER SET utf8;
CREATE USER 'hive'@'%'IDENTIFIED BY 'Kylin@2021!';
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%' IDENTIFIED BY 'Kylin@2021!'; 
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' IDENTIFIED BY 'Kylin@2021!' WITH GRANT OPTION;
-- hue
CREATE DATABASE hue DEFAULT CHARACTER SET utf8;
CREATE USER 'hue'@'%'IDENTIFIED BY 'Kylin@2021!';
GRANT ALL PRIVILEGES ON hue.* TO 'hue'@'%' IDENTIFIED BY 'Kylin@2021!'; 
GRANT ALL PRIVILEGES ON *.* TO 'hue'@'%' IDENTIFIED BY 'Kylin@2021!' WITH GRANT OPTION;
-- oozie
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8;
CREATE USER 'oozie'@'%'IDENTIFIED BY 'Kylin@2021!';
GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'Kylin@2021!'; 
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%' IDENTIFIED BY 'Kylin@2021!' WITH GRANT OPTION;
-- 刷新mysql的权限列表 
FLUSH PRIVILEGES;

• 主节点放置Mysql驱动包

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

3. 安装依赖

• 安装jdk, 配置Java home

yum install java-1.8.0-openjdk-devel.x86_64
ll /etc/alternatives/java

• 安装依赖

yum install -y bind-utils psmisc libxslt cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs httpd mod_ssl openssl-devel /lib/lsb/init-functions libpq.so.5python27postgresql-devel*postgresql-odbc.x86_64python2-develchkconfig zlib sqlite  redhat-lsb postgresql*   openssl  telnet pcre-devel gcc gcc-c++ MySQL-python 

• 更新pip

pip install --upgrade pip
pip install --upgrade setuptools
pip2 install psycopg2 

4. 部署CDH

CDH版本6.3.2, 官网停止下载了, 我还留了一份安装包, csdn下载地址

安装CM

• 解压, 主节点安装

cd /opt/cloudera-manager/cm6.3.1/RPMS/x86_64/
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

• 其他节点

cd /opt/cloudera-manager/cm6.3.1/RPMS/x86_64/
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

• 所有节点修改agent配置, 指向master

/etc/cloudera-scm-agent/config.ini server_host=主机名

• 初始化数据库, 测试Mysql连接

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm Kylin@2021!
All done, your SCM database is configured correctly! 成功

• 启动cm server

# 主节点启动server
systemctl restart cloudera-scm-server
# 查看server运行状态
service cloudera-scm-server status
# 查看server日志
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
# 所有节点启动agent
systemctl restart cloudera-scm-agent  
# 查看agent运行状态
service cloudera-scm-agent  status
# 查看agent日志
tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

• 启动完毕, 浏览器访问master ip:7180, 默认密码admin/admin

安装CDH

• 开始离线安装CDH,准备parcel包

# 部署离线parcel源
mkdir -p /var/www/html/cdh6_parcel
# 将parcel包放在httpd的目录下
cp cdh/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel /var/www/html/cdh6_parcel/
cp cdh/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1 /var/www/html/cdh6_parcel/CDH-6.3.1
cp cdh/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1 /var/www/html/cdh6_parcel/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha
cp cdh/manifest.json /var/www/html/cdh6_parcel/
# 启动httpd服务
systemctl start httpd

浏览器访问master ip/cdh6_parcel

• 返回CM, 选择免费版下一步下一步, 选主机, 更多选项, 添加源继续, 等解压, 检查集群, 忽略,下一步, 选择安装的组件, 使用之前创建好的账号配置数据库, 配置默认下一步
• CDH部署完成

5. 部署Kylin

以上都顺利的话, 按照官方步骤执行, 即可启动成功
Deploy Kylin4 on CDH6