官方网站
https://docs.cloudera.com/documentation/enterprise/6/6.3.html
https://archive.cloudera.com/phoenix/
https://archive.cloudera.com/cdh6/
https://archive.cloudera.com/cm6/
需下载文件:
提示:该方案是在离线的情况下安装
目录
前言
提示:CDH收购HDP后进行了相关的优化,现在CDH变得更加主流
一、Hadoop的3种发行版:
Apache Hadoop
Cloudera’s Distribution Including Apache Hadoop(CDH)
Hortonworks Data Platform (HDP)
二、操作步骤
1.修改ip
ip addr 查看第二个 (ifcfg-eth0)
vi /etc/sysconfig/network-scripts/ifcfg-eth0(不确定)
BOOTPROTO=static 静态ip
ONBOOT=yes
IPADDR=127.0.0.1 ip地址
NETMASK=255.255.255.0 子网掩码
GATEWAY=127.0.0.1 网关
重启服务 service network restart
2.修改hosts文件
vi /etc/hosts
127.0.0.88 cdh-001
127.0.0.89 cdh-002
127.0.0.90 cdh-003
127.0.0.91 cdh-004
127.0.0.92 cdh-005
127.0.0.93 cdh-006
3.关闭防火墙(所有机器)
systemctl stop firewalld
systemctl disable firewalld
4.关闭selinux(所有机器)
vi /etc/selinux/config
将SELINUX=enforcing改为SELINUX=disabled
设置后需要重启才能生效
4.安装相关服务
yum install -y httpd //安装cdh需要使用 (主服务器)
yum install -y perl //安装cdh需要使用(所有服务器)
如果安装失败就挂载本地yum源
mkdir /home/iso
mount -t iso9660 -o loop /mysoft/CentOS-7-x86_64-DVD-1804.iso /home/iso
cd /etc/yum.repos.d/
注意要把没有用到的.repo中每个文件的备份一下
mv CentOS-Base.repo CentOS-Base.repo.bak
cp CentOS-CR.repo CentOS-Local.repo
vi CentOS-Local.repo(编辑文件)
设置enabled=0
baseurl=file:///home/iso
#启动yum源: 1-启用 0-不启用
enabled=1
#安全检测: 1-开启 0-不开启
gpgcheck=1
gpgkey=file:///home/iso/RPM-GPG-KEY-CentOS-7
更新缓存
yum clean all
yum update
yum makecache
5.安装jdk
mkdir /usr/java
tar -zxvf jdk-8u45-linux-x64.gz -C /usr/java/
#切记必须修正所属用户及用户组
chown -R root:root /usr/java/jdk1.8.0_45
vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=${JAVA_HOME}/bin:${PATH}
source /etc/profile
6.安装mysql
1.解压
tar -zxvf mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz -C /usr/local
cd /usr/local/
mv mysql-5.7.11-linux-glibc2.5-x86_64 mysql
mkdir mysql/arch mysql/data mysql/tmp
mv /etc/my.cnf /etc/my.cnf.bak
2.创建my.cnf
vi /etc/my.cnf
[client]
port = 3306
socket = /usr/local/mysql/data/mysql.sock
default-character-set=utf8mb4
[mysqld]
port = 3306
socket = /usr/local/mysql/data/mysql.sock
skip-slave-start
skip-external-locking
key_buffer_size = 256M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
query_cache_size= 32M
max_allowed_packet = 16M
myisam_sort_buffer_size=128M
tmp_table_size=32M
table_open_cache = 512
thread_cache_size = 8
wait_timeout = 86400
interactive_timeout = 86400
max_connections = 600
#Try number of CPU's*2 for thread_concurrency
#thread_concurrency = 32
#isolation level and default engine
default-storage-engine = INNODB
transaction-isolation = READ-COMMITTED
server-id = 1739
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
pid-file = /usr/local/mysql/data/hostname.pid
#open performance schema
log-warnings
sysdate-is-now
binlog_format = ROW
log_bin_trust_function_creators=1
log-error = /usr/local/mysql/data/hostname.err
log-bin = /usr/local/mysql/arch/mysql-bin
expire_logs_days = 7
innodb_write_io_threads=16
relay-log = /usr/local/mysql/relay_log/relay-log
relay-log-index = /usr/local/mysql/relay_log/relay-log.index
relay_log_info_file= /usr/local/mysql/relay_log/relay-log.info
log_slave_updates=1
gtid_mode=OFF
enforce_gtid_consistency=OFF
#slave
slave-parallel-type=LOGICAL_CLOCK
slave-parallel-workers=4
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON
#other logs
#general_log =1
#general_log_file = /usr/local/mysql/data/general_log.err
#slow_query_log=1
#slow_query_log_file=/usr/local/mysql/data/slow_log.err
#for replication slave
sync_binlog = 500
#for innodb options
innodb_data_home_dir = /usr/local/mysql/data/
innodb_data_file_path = ibdata1:1G;ibdata2:1G:autoextend
innodb_log_group_home_dir = /usr/local/mysql/arch
innodb_log_files_in_group = 4
innodb_log_file_size = 1G
innodb_log_buffer_size = 200M
#根据生产需要,调整pool size
innodb_buffer_pool_size = 2G
#innodb_additional_mem_pool_size = 50M #deprecated in 5.6
tmpdir = /usr/local/mysql/tmp
innodb_lock_wait_timeout = 1000
#innodb_thread_concurrency = 0
innodb_flush_log_at_trx_commit = 2
innodb_locks_unsafe_for_binlog=1
#innodb io features: add for mysql5.5.8
performance_schema
innodb_read_io_threads=4
innodb-write-io-threads=4
innodb-io-capacity=200
#purge threads change default(0) to 1 for purge
innodb_purge_threads=1
innodb_use_native_aio=on
#case-sensitive file names and separate tablespace
innodb_file_per_table = 1
lower_case_table_names=1
[mysqldump]
quick
max_allowed_packet = 128M
[mysql]
no-auto-rehash
default-character-set=utf8mb4
[mysqlhotcopy]
interactive-timeout
[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M
3:创建用户
groupadd -g 101 dba
useradd -u 514 -g dba -G root -d /usr/local/mysql mysql
4.copy 环境变量配置文件至mysql用户的home目录中,为了以下步骤配置个人环境变量
cp /etc/skel/.* /usr/local/mysql ###important (切换用户su - mysql) 出现的-bash-$问题的解决
5.配置环境变量
vi /usr/local/mysql/.bashrc
export MYSQL_BASE=/usr/local/mysql
export PATH=${MYSQL_BASE}/bin:$PATH
6.赋权限和用户组,安装
[root@cdh-tpf001 local]# chown mysql:dba /etc/my.cnf //设置所有者
[root@cdh-tpf001 local]# chmod 640 /etc/my.cnf //设置文件的权限(6:所有者拥有读写,同组拥有读,其他没有权限)
[root@cdh-tpf001 local]# chown -R mysql:dba /usr/local/mysql
[root@cdh-tpf001 local]# chmod -R 775 /usr/local/mysql
7.配置服务及开机自启动
[root@cdh-tpf001 local]# cd /usr/local/mysql
#将服务文件拷贝到init.d下,并重命名为mysql
[root@cdh-tpf001 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysql
#赋予可执行权限
[root@cdh-tpf001 mysql]# chmod +x /etc/rc.d/init.d/mysql
#添加服务
[root@hadoop39 mysql]# chkconfig --add mysql
8.初始化
bin/mysqld \
--defaults-file=/etc/my.cnf \
--user=mysql \
--basedir=/usr/local/mysql/ \
--datadir=/usr/local/mysql/data/ \
--initialize
9.查看临时密码(在data目录下面)
cd /usr/local/mysql/data
cat hostname.err |grep password
出现:
[Note] A temporary password is generated for root@localhost: ebBL3#OJu-C
10.启动:
su - mysql//切换用户
service mysql start //启动
11.登录及修改用户密码
/usr/local/mysql/data:>mysql -uroot -p
输入上面查看的密码
登陆出现:ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
在my.cnf中的[mysqld]后面任意一行添加“skip-grant-tables”用来跳过密码验证的过程
service mysqld restart
然后直接输入mysql就能进入 执行 flush privileges;
修改密码
mysql> alter user root@localhost identified by '密码';
quit退出,然后修改my.cnf文件,去掉skip-grant-tables
mysql> alter user root@localhost identified by '密码';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '密码' ;
//cm启动需要存储的元数据
create database cmf DEFAULT CHARACTER SET utf8;
//cdh启动需要存储的元数据
create database amon DEFAULT CHARACTER SET utf8;
grant all on cmf.* TO 'cmf'@'%' IDENTIFIED BY '密码';
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY '密码';
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> quit;
12:部署mysql驱动(/etc/cloudera-scm-agent/config.ini)
mkdir -p /usr/share/java/
cp /mysoft/mysql-connector-java-5.1.47.jar /usr/share/java/
cd /usr/share/java/
mv mysql-connector-java-5.1.47.jar mysql-connector-java.jar
7.ntp时间同步
安装 ntp
yum install -y ntp (所有节点都要安装)
编辑配置文件
vi /etc/ntp.conf
//当我们的外部时钟不能使用的时候,使用的是本地的时间
server 127.127.1.0 iburst local clock
//时钟同步老大允许哪个网段的小弟来同步我们的时间,此处是内网,
//10.80.4.0来表示这个网段,不行再进行测试,nomodify是指允许哪些节点
//来访问的;不行的话再进行调整。
restrict 10.80.4.0 mask 255.255.255.0 nomodify
开启服务:systemctl start ntpd
查看服务状态:systemctl status ntpd 显示active running就是okay的状态
查看网段下时间的偏移: ntpq -p
时区同步命令(其它机器上执行)
次节点:(依次执行下面命令)
systemctl stop ntpd
systemctl disable ntpd
ntpdate cdh-tpf001
crontab -e
添加
# 分 时 日 月 周,每天00:00执行该脚本
00 00 * * * /usr/sbin/ntpdate cdh-tpf001
要将时区设置成一样的 (如果时区一致,就不需要以下设置)
date
timedatectl --help
timedatectl set-timezone Asia/Shanghai
8.rpm包部署CM
主节点:
cd /mysoft
tar -zxvf cm6.3.1-redhat7.tar.gz
cd /mysoft/cm6.3.1/RPMS/x86_64
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
cd /etc/cloudera-scm-server/
vi db.properties
修改内容:
com.cloudera.cmf.db.host=cdh-tpf001:3306
com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.password=cmf
com.cloudera.cmf.db.setupType=EXTERNAL
启动cm
service cloudera-scm-server start
/var/log/cloudera-scm-server/cloudera-scm-server.log
service cloudera-scm-server status
cat /var/log/cloudera-scm-server/cloudera-scm-server.log
tail -F /var/log/cloudera-scm-server/cloudera-scm-server.log 查看启动过程
启动完成后:
http://cdh-tpf001:7180/cmf/login (此时不用操作任何东西进行如下操作可以免去免秘登陆)
从节点
cd /mysoft/cm6.3.1/RPMS/x86_64
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
vi /etc/cloudera-scm-agent/config.ini
service cloudera-scm-agent start(需要安装组件的地方都有执行)
9.配置cdh pracel
mkdir /var/www/html/cdh6_parcel
cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /var/www/html/cdh6_parcel
cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 /var/www/html/cdh6_parcel
cp manifest.json /var/www/html/cdh6_parcel/
cd /var/www/html/cdh6_parcel/
mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
systemctl start httpd.service #启动
http://cdh-tpf001/cdh6_parcel/ 查看一下是否httpd正常
10.配置phoenix pracel
mkdir /var/www/html/cdh6_phoenix
cp PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel /var/www/html/cdh6_phoenix/
cp PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel.sha /var/www/html/cdh6_phoenix/
cp manifest-phoenix.json /var/www/html/cdh6_phoenix/
cp PHOENIX-1.0.jar /opt/cloudera/csd
cd /var/www/html/cdh6_phoenix
mv manifest-phoenix.json manifest.json
http://cdh-tpf001/cdh6_phoenix/ 查看一下是否httpd正常
11.安装CDH
step 1. 网页打开 cdh-tpf001:7180,登录(admin/admin)(选择免费)
step 2.点击下一步
step 3. 集群名称,此处根据实际命名
step 4. 指定主机
step 5. 选择存储库(上面的http pracel服务的位置)
step 6. 自动刷新页面,显示CDH
step 7. 安装Parcels
step 8. 检查群集
修复出现的问题
修复透明大页面(主、次)
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
swappiness 设置(主、次)
sysctl vm.swappiness=10
echo ‘vm.swappiness=10’>> /etc/sysctl.conf
修复Psycopg2版本较低(忽略)
step 9. 服务选择
step 10. 角色分配
step 11. 数据库设置,测试连接(上面创建的amon数据库)
step 12. 首次运行命令详细信息
step 13. 汇总
5.5 安装成功(可以修改标黄的问题,有的也可以选择忽略)
12 Phoenix安装
step1: 并将PHOENIX-1.0.jar文件复制到/opt/cloudera/csd目录下(上面已经完成忽略)
step2: 设置parcel
step3: 添加服务
step4: 配置hbase-site.xml
<property>
<name>hbase.regionserver.wal.codec</name>
<value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
</property>
<property>
<name>phoenix.functions.allowUserDefinedFunctions</name>
<value>true</value>
<description>enable UDF functions</description>
</property>
总结
如果有数据库,忽略数据库的安装
如果上面安装phoenix服务仍然没有
执行以下操作:
systemctl restart cloudera-scm-server
在web页面操作 cloudera Management service 重启
测试phoenix:
cd /opt/cloudera/parcels/PHOENIX/bin
./phoenix-sqlline localhost:2181/hbase
创建表:
CREATE TABLE IF NOT EXISTS my_test (
state CHAR(2) NOT NULL,
city VARCHAR NOT NULL,
population BIGINT
CONSTRAINT my_pk PRIMARY KEY (state, city));
插入数据
upsert into my_test values('01','shanghai',10);
创建索引
create index TEST_IDX on my_test(state);
查看执行计划,只要没有Full SCAN 就说明创建成功
explain select state from my_test where state='01';
附加
如果重装集群的话
1: 首先要关闭集群
2:要清空下面文件夹:
/dfs
/tmp
3:master启动不了
WARN [master/s110:16000:becomeActiveMaster] master.HMaster: hbase:meta,,1.1588230740 is NOT online;
要清空zookeeper
cd /opt/cloudera/parcels/CDH/bin/
./zookeeper-client -server localhost:2181
ls /hbase
rmr /hbase
4:如果cm在安装集群的时候已经指定了数据库,再次安装的时候不需要指定了
5:Activity monitory就是指定数据库
6:安装的时候指定的hdfs的nanmenode好datanode文件夹不能存在
7 : spark-default.xml配置(用于提交Spark程序)
spark.executor.extraClassPath = /opt/cloudera/parcels/PHOENIX/lib/phoenix/*
spark.driver.extraClassPath = /opt/cloudera/parcels/PHOENIX/lib/phoenix/*
日志文件目录调整