Centos6.9 下的 CM（Cloudera Manager）&CDH 大数据环境部署

最新推荐文章于 2024-01-04 22:10:43 发布

初见丶晨曦

最新推荐文章于 2024-01-04 22:10:43 发布

阅读量872

点赞数

分类专栏： CDH大数据环境搭建

本文链接：https://blog.csdn.net/kewanjun_lcx/article/details/80319445

版权

CDH大数据环境搭建专栏收录该内容

1 篇文章 0 订阅

订阅专栏

本文档详细记录了在CentOS环境下搭建Hadoop集群的过程，包括JDK、MySQL等软件的安装配置，NTP时间同步，Cloudera Manager Server与Agent的部署，以及Hive、Oozie等组件的具体配置步骤。

摘要由CSDN通过智能技术生成

（系统采用CentOS-6.9-x86_64-bin-DVD1.iso镜像，百度下载方式）
环境采用的是一台物理机ESXI 虚拟镜像系统下三台虚拟机 n1、n2、n3 （n1 是master&salver，n2&n3 is salver）
无特殊说明以下操作配置在每台机器都要操作

一、基本环境安装

1.JDK安装

#java -version
#tar xzvf jdk-8u111-linux-x64.gz
#vim /etc/profile export JAVA_HOME=/usr/local/java export PATH=$JAVA_HOME/bin:$PATH export
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
#source /etc/profile
#java -version

2.修改Linux机器的网络自动连接

#cd /etc/sysconfig/network-scripts/
#vi ifcfg-eth0
#ONBOOT="yes" (内网IP不通,修改BOOTPROTO=static)[修改为静态IP 否则IP设置不起作用]

3.配置HOSTNAME & 配置HOSTS

>  vi /etc/sysconfig/network
>  HOSTNAME=n1#其他依次改为 n2 n3
> :wq
>  vi /etc/hosts
> 192.168.1.181 n1
> 192.168.1.182 n2
> 192.168.1.183 n3
> :wq
> #hostname 如果未改动，重启机器

4.SSH免密钥登录

> # ssh-keygen -t rsa 一直回车
> # cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
> # chmod 600 ~/.ssh/authorized_keys
> # scp ~/.ssh/authorized_keys root@n2:~/.ssh/
> # scp ~/.ssh/authorized_keys root@n3:~/.ssh/
> # ssh-copy-id -i root@n1
> # ssh-copy-id -i root@n2
> # ssh-copy-id -i root@n3

5.防火墙关闭(此处centos7 以上版本不同另行百度)

> # service iptables stop
> # chkconfig iptables off

6.SELLINUX关闭

> # set enforce 0
> # vi /etc/selinux/config
> # SELINUX=disabled

7.安装JDK配置环境变量

> # scp jdk-7u80-linux-x64.tar.gz  node1:/opt(复制到node1)
> # scp jdk-7u80-linux-x64.tar.gz  node2:/opt(复制到node2)

8.Ntp时间配置
离线安装准备rpm如下（下载认准.2. 其他我没用起来）:
ntp-4.2.6p5-10.el6.centos.2.x86_64.rpm
ntpdate-4.2.6p5-10.el6.centos.2.x86_64.rpm
1.配置内网NTP-Server
①修改时区（三台机器的时区一定要一致）

# vi /etc/sysconfig/clock ZONE="Asia/Shanghai" UTC=false ARC=false 
(后置事项: 所有机器date命令查询时间，确保时间CST时区，不是的操作如下： 命令tzselect，时区调到shanghai)
> # source /etc/sysconfig/clock

②安装rpm（服务器、客户端机器全部相同安装）
③修改配置文件 /etc/ntp.conf
A.服务器配置

#建议直接删除原来文件 直接拷贝 避免出错
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1

# Hosts on local network are less restricted.
# 允许内网其他机器同步时间
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
# 中国这边最活跃的时间服务器 : http://www.pool.ntp.org/zone/cn
server 210.72.145.44 perfer   # 中国国家受时中心
server 202.112.10.36             # 1.cn.pool.ntp.org
server 59.124.196.83             # 0.asia.pool.ntp.org

#broadcast 192.168.1.255 autokey        # broadcast server
#broadcastclient                        # broadcast client
#broadcast 224.0.1.1 autokey            # multicast server
#multicastclient 224.0.1.1              # multicast client
#manycastserver 239.255.254.254         # manycast server
#manycastclient 239.255.254.254 autokey # manycast client

# allow update time by the upper server 
# 允许上层时间服务器主动修改本机时间
restrict 210.72.145.44 nomodify notrap noquery
restrict 202.112.10.36 nomodify notrap noquery
restrict 59.124.196.83 nomodify notrap noquery

# Undisciplined Local Clock. This is a fake driver intended for backup
# and when no outside source of synchronized time is available. 
# 外部时间服务器不可用时，以本地时间作为时间服务
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 10

# Enable public key cryptography.
#crypto

includefile /etc/ntp/crypto/pw

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography. 
keys /etc/ntp/keys

# Specify the key identifiers which are trusted.
#trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8

# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats

B.客户端配置

#建议直接删除原来文件 直接拷贝 避免出错
driftfile /var/lib/ntp/drift
restrict 127.0.0.1
restrict -6 ::1

# 配置时间服务器为本地的时间服务器
server 192.168.1.135

restrict 192.168.1.135 nomodify notrap noquery

server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 10

includefile /etc/ntp/crypto/pw

keys /etc/ntp/keys

2.启动ntp服务 service ntpd start chkconfig ntpd on service ntpd status

其他客户端机器上操作如下指令 强制 同步时间
#  ntpdate -u 192.168.1.181
启动后，查看同步情况 指令
# ntpq -p
# ntpstat

3 错误问题 
 客户端连接服务端失败，查找不到，尝试123端口加入防火墙规格或者关闭防火墙！

9.MySQL安装配置（元数据只在 master 端安装即可）

> 如果你的centos 无 perl解释器 ，先安装perl ，二进制编译安装
> 补：(有些linux上未安装Perl编译器、安装MySQL出错,Perl源码包需要编译安装)
> 1.检查gcc  gcc -v ①有版本跳过 ②准备如下rpm，从 
> http://mirrors.163.com/centos/6/os/x86_64/Packages/
> http://mirrors.aliyun.com/centos/7/os/x86_64/Packages/ 
**> 下载
> mpfr-3.1.1-4.el7.x86_64.rpm libmpc-1.0.1-3.el7.x86_64.rpm
> kernel-headers-3.10.0-123.el7.x86_64.rpm
> glibc-headers-2.17-55.el7.x86_64.rpm
> glibc-devel-2.17-55.el7.x86_64.rpm cpp-4.8.2-16.el7.x86_64.rpm**
> gcc-4.8.2-16.el7.x86_64.rpm ③rpm -Uvh *.rpm --nodeps --force ④gcc -v
> 检查 http://www.perl.org/get.html下载perl源码包
> 2.tar解压 进入执行 ./Configure -des -Dprefix=/usr/local/perl -Dusethreads -Uversiononly （先mkdir /usr/local/perl）
> 3.完成后make  然后 make install
> 4.ln -s /usr/local/perl/bin/perl /usr/bin/perl (建立软连接)
> 5.perl -v 检查安装结果[Mysql安装依赖问题]

①在没有Perl-**一堆依赖的机器离线采用二进制安装文件 mysql-5.6.40-linux-glibc2.12-x86_64.tar.gz （注意版本号）
②建组、建用户

groupadd mysql
useradd -g mysql mysql
解压安装程序：
tar -zxvf mysql.tar.gz
mv /rpm/mysql /usr/local/mysql
初始化：
./scripts/mysql_install_db --basedir=/usr/local/mysql/ --datadir=/usr/local/mysql/data/ --user=mysql
配置文件：
cp /usr/local/mysql/support-files/my-default.cnf /etc/my.cnf
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
port = 3306
log_error=mysql.err
修改目录权限：
chown -R mysql:mysql /usr/local/mysql/
修改环境变量：
Vi /etc/profile
export PATH=/usr/local/mysql/bin:$PATH
设置mysql开机自动启动：
cp /usr/local/mysql/support-files/mysql.server /etc/rc.d/init.d/mysqld
chown -R root:root /etc/rc.d/init.d/mysqld
chmod 755 /etc/rc.d/init.d/mysqld
chkconfig --add mysqld
chkconfig --list mysqld
mysqld          0:off 1:off 2:on 3:on 4:on 5:on 6:off
service mysqld status
ERROR! MySQL is not running

③创建所需数据库（整个操作都是修改mysql数据库的user表可以用sql操作）

#mysql
#update user set password=password("youotech") where user='root';
# GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'youotech' WITH GRANT OPTION;//   修改登陆数据库的权限
（set password for 'root'@'%'=password('oracle');）
mysql的root用户授权：
mysql> grant all privileges on *.* to 'root'@'%' identified by 'oracle';
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> set password=password('oracle');//修改密码
Query OK, 0 rows affected (0.00 sec)
# flush privileges;//刷新权限
创建4个数据库hive、scm和oozie 、sentry，创建用户hadoop并授权，密码默认为oracle（根据需要，自己修改），执行如下命令
# create database hive default charset utf8 collate utf8_general_ci;
**（hive 不可以用 utf-8编码，后续建表会出现字长错误 无法插入元数据）**
# create database scm default charset utf8 collate utf8_general_ci;
# create database sentry default charset utf8 collate utf8_general_ci;
# create database oozie default charset utf8 collate utf8_general_ci;
# CREATE USER 'hadoop'@'%' IDENTIFIED BY 'oracle';
# CREATE USER 'hadoop'@'localhost' IDENTIFIED BY 'oracle';
# GRANT ALL PRIVILEGES ON *.* TO 'hadoop'@'%' IDENTIFIED BY 'oracle';
# GRANT ALL PRIVILEGES ON *.* TO 'hadoop'@'localhost' IDENTIFIED BY 'oracle';
# flush privileges;[Perl依赖不全的机器采用二进制安装，需采用perl源码编译安装后。]

Mysql 忘记操作:
# /etc/init.d/mysql stop
# mysqld_safe --user=mysql --skip-grant-tables --skip-networking &
# mysql -u root
mysql> update user set password=password("newpassword") where user='root';
mysql> flush privileges;
mysql> quit
# /etc/init.d/mysql restart
# mysql -uroot -p
enter password: <输入新设的密码newpassword>

二、Cloudera Manager 安装

所有机器都需要安装配置

1.安装Cloudera-Manager Server、Agent

 #cd /opt
 #mkdir cloudera-manager
 #tar -xvzf cloudera-manager-el6-cm5.7.0_x86_64.tar.gz  -C /opt/cloudera-manager

2.创建用户Cloudera- # useradd –system –no-create-home –shell=/bin/false –comment “Cloudera SCM User” cloudera-scm置CM Agent （从机配置）

# cd /opt/cloudera-manager/cm-5.7.0/etc/cloudera-scm-agent
# vim config.ini
# server_host=192.168.1.210
同步Agent到其他节点
scp -r /opt/cloudera-manager/cm-5.7.0 root@n2:/opt/cloudera-manager/

4.配置CM SERVER 数据库（主机配置）

拷贝mysql-connector-java-6.0.6-bin.jar 文件到目录 /usr/share/java/
注意修改jar包名称要修改为mysql-connector-java.jar
# mkdir /usr/share/java
# mv mysql-connector-java-6.0.6-bin.jar mysql-connector-java.jar
mysql-connector-java-6.0.6-bin.jar在mysql-connector-java-6.0.6.zip的压缩包里面
# mysql -u root -p
#grant all privileges on *.* to 'temp'@'%' identified by 'temp' with grant option; 
# cd /opt/cloudera-manager/cm-5.7.0/share/cmf/schema/
# ./scm_prepare_database.sh mysql temp -h node0 -utemp -ptemp --scm-host node0 scm scm scm
(./scm_prepare_database.sh mysql cm -h n1 -uroot -p***** --scm-host n1 scm scm scm)

5.创建Parcel目录

SERVER 节点
# mkdir -p /opt/cloudera/parcel-repo
# chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
Agent节点
# mkdir -p /opt/cloudera/parcels
# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

6.制作CDH本地源

# cd /opt/cloudera/parcel-repo/
# cp /opt/CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parce* ./
# cp /opt/manifest.json  ./

7.启动CM Server、Agent

# cd /opt/cloudera-manager/cm-5.7.0/etc/init.d/
# ./cloudera-scm-server start
# ./cloudera-scm-agent start
如果# ./cloudera-scm-agent start失败
# cd ../..
# cd log
# cd cloudera-scm-agent/
# tail -f cloudera-scm-agent.log
(tail -f cloudera-scm-agent.out)
报Unable to create the pidfile错误的时候，请按一下步骤操作:
# cd /opt/cloudera-manager/cm-5.7.0/run/
# mkdir cloudera-scm-agent
# cd /opt/cloudera-manager/cm-5.7.0/etc/init.d/
# ./cloudera-scm-agent start
成功解决./cloudera-scm-agent start失败的错误

8.启动CM后台管理
http://192.168.1.*:7180
admin-admin 初始账号密码

常见错误汇总(所有我遇到的)：

错误信息解决:
1.Table 'oozie.VALIDATE_CONN' doesn't exist：
手动创建sql


DROP TABLE IF EXISTS `BUNDLE_ACTIONS`;

CREATE TABLE `BUNDLE_ACTIONS` (
  `bundle_action_id` varchar(255) NOT NULL,
  `bundle_id` varchar(255) DEFAULT NULL,
  `coord_id` varchar(255) DEFAULT NULL,
  `coord_name` varchar(255) DEFAULT NULL,
  `critical` int(11) DEFAULT NULL,
  `last_modified_time` datetime DEFAULT NULL,
  `pending` int(11) DEFAULT NULL,
  `status` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`bundle_action_id`),
  KEY `I_BNDLTNS_BUNDLE_ID` (`bundle_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

DROP TABLE IF EXISTS `BUNDLE_JOBS`;

CREATE TABLE `BUNDLE_JOBS` (
  `id` varchar(255) NOT NULL,
  `app_name` varchar(255) DEFAULT NULL,
  `app_path` varchar(255) DEFAULT NULL,
  `conf` mediumblob,
  `created_time` datetime DEFAULT NULL,
  `end_time` datetime DEFAULT NULL,
  `external_id` varchar(255) DEFAULT NULL,
  `group_name` varchar(255) DEFAULT NULL,
  `job_xml` mediumblob,
  `kickoff_time` datetime DEFAULT NULL,
  `last_modified_time` datetime DEFAULT NULL,
  `orig_job_xml` mediumblob,
  `pause_time` datetime DEFAULT NULL,
  `pending` int(11) DEFAULT NULL,
  `start_time` datetime DEFAULT NULL,
  `status` varchar(255) DEFAULT NULL,
  `suspended_time` datetime DEFAULT NULL,
  `time_out` int(11) DEFAULT NULL,
  `time_unit` varchar(255) DEFAULT NULL,
  `user_name` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `I_BNDLJBS_CREATED_TIME` (`created_time`),
  KEY `I_BNDLJBS_LAST_MODIFIED_TIME` (`last_modified_time`),
  KEY `I_BNDLJBS_STATUS` (`status`),
  KEY `I_BNDLJBS_SUSPENDED_TIME` (`suspended_time`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


DROP TABLE IF EXISTS `COORD_ACTIONS`;

CREATE TABLE `COORD_ACTIONS` (
  `id` varchar(255) NOT NULL,
  `action_number` int(11) DEFAULT NULL,
  `action_xml` mediumblob,
  `console_url` varchar(255) DEFAULT NULL,
  `created_conf` mediumblob,
  `created_time` datetime DEFAULT NULL,
  `error_code` varchar(255) DEFAULT NULL,
  `error_message` varchar(255) DEFAULT NULL,
  `external_id` varchar(255) DEFAULT NULL,
  `external_status` varchar(255) DEFAULT NULL,
  `job_id` varchar(255) DEFAULT NULL,
  `last_modified_time` datetime DEFAULT NULL,
  `missing_dependencies` mediumblob,
  `nominal_time` datetime DEFAULT NULL,
  `pending` int(11) DEFAULT NULL,
  `push_missing_dependencies` mediumblob,
  `rerun_time` datetime DEFAULT NULL,
  `run_conf` mediumblob,
  `sla_xml` mediumblob,
  `status` varchar(255) DEFAULT NULL,
  `time_out` int(11) DEFAULT NULL,
  `tracker_uri` varchar(255) DEFAULT NULL,
  `job_type` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `I_CRD_TNS_CREATED_TIME` (`created_time`),
  KEY `I_CRD_TNS_EXTERNAL_ID` (`external_id`),
  KEY `I_CRD_TNS_JOB_ID` (`job_id`),
  KEY `I_CRD_TNS_LAST_MODIFIED_TIME` (`last_modified_time`),
  KEY `I_CRD_TNS_NOMINAL_TIME` (`nominal_time`),
  KEY `I_CRD_TNS_RERUN_TIME` (`rerun_time`),
  KEY `I_CRD_TNS_STATUS` (`status`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


DROP TABLE IF EXISTS `COORD_JOBS`;

CREATE TABLE `COORD_JOBS` (
  `id` varchar(255) NOT NULL,
  `app_name` varchar(255) DEFAULT NULL,
  `app_namespace` varchar(255) DEFAULT NULL,
  `app_path` varchar(255) DEFAULT NULL,
  `bundle_id` varchar(255) DEFAULT NULL,
  `concurrency` int(11) DEFAULT NULL,
  `conf` mediumblob,
  `created_time` datetime DEFAULT NULL,
  `done_materialization` int(11) DEFAULT NULL,
  `end_time` datetime DEFAULT NULL,
  `execution` varchar(255) DEFAULT NULL,
  `external_id` varchar(255) DEFAULT NULL,
  `frequency` varchar(255) DEFAULT NULL,
  `group_name` varchar(255) DEFAULT NULL,
  `job_xml` mediumblob,
  `last_action_number` int(11) DEFAULT NULL,
  `last_action` datetime DEFAULT NULL,
  `last_modified_time` datetime DEFAULT NULL,
  `mat_throttling` int(11) DEFAULT NULL,
  `next_matd_time` datetime DEFAULT NULL,
  `orig_job_xml` mediumblob,
  `pause_time` datetime DEFAULT NULL,
  `pending` int(11) DEFAULT NULL,
  `sla_xml` mediumblob,
  `start_time` datetime DEFAULT NULL,
  `status` varchar(255) DEFAULT NULL,
  `suspended_time` datetime DEFAULT NULL,
  `time_out` int(11) DEFAULT NULL,
  `time_unit` varchar(255) DEFAULT NULL,
  `time_zone` varchar(255) DEFAULT NULL,
  `user_name` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `I_CRD_JBS_BUNDLE_ID` (`bundle_id`),
  KEY `I_CRD_JBS_CREATED_TIME` (`created_time`),
  KEY `I_CRD_JBS_END_TIME` (`end_time`),
  KEY `I_CRD_JBS_LAST_MODIFIED_TIME` (`last_modified_time`),
  KEY `I_CRD_JBS_NEXT_MATD_TIME` (`next_matd_time`),
  KEY `I_CRD_JBS_STATUS` (`status`),
  KEY `I_CRD_JBS_SUSPENDED_TIME` (`suspended_time`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


DROP TABLE IF EXISTS `OOZIE_SYS`;

CREATE TABLE `OOZIE_SYS` (
  `name` varchar(100) DEFAULT NULL,
  `data` varchar(100) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


DROP TABLE IF EXISTS `OPENJPA_SEQUENCE_TABLE`;

CREATE TABLE `OPENJPA_SEQUENCE_TABLE` (
  `ID` tinyint(4) NOT NULL,
  `SEQUENCE_VALUE` bigint(20) DEFAULT NULL,
  PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


DROP TABLE IF EXISTS `SLA_EVENTS`;

CREATE TABLE `SLA_EVENTS` (
  `event_id` bigint(20) NOT NULL AUTO_INCREMENT,
  `alert_contact` varchar(255) DEFAULT NULL,
  `alert_frequency` varchar(255) DEFAULT NULL,
  `alert_percentage` varchar(255) DEFAULT NULL,
  `app_name` varchar(255) DEFAULT NULL,
  `dev_contact` varchar(255) DEFAULT NULL,
  `group_name` varchar(255) DEFAULT NULL,
  `job_data` text,
  `notification_msg` text,
  `parent_client_id` varchar(255) DEFAULT NULL,
  `parent_sla_id` varchar(255) DEFAULT NULL,
  `qa_contact` varchar(255) DEFAULT NULL,
  `se_contact` varchar(255) DEFAULT NULL,
  `sla_id` varchar(255) DEFAULT NULL,
  `upstream_apps` text,
  `user_name` varchar(255) DEFAULT NULL,
  `bean_type` varchar(31) DEFAULT NULL,
  `app_type` varchar(255) DEFAULT NULL,
  `event_type` varchar(255) DEFAULT NULL,
  `expected_end` datetime DEFAULT NULL,
  `expected_start` datetime DEFAULT NULL,
  `job_status` varchar(255) DEFAULT NULL,
  `status_timestamp` datetime DEFAULT NULL,
  PRIMARY KEY (`event_id`),
  KEY `I_SL_VNTS_DTYPE` (`bean_type`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


DROP TABLE IF EXISTS `SLA_REGISTRATION`;

CREATE TABLE `SLA_REGISTRATION` (
  `job_id` varchar(255) NOT NULL,
  `app_name` varchar(255) DEFAULT NULL,
  `app_type` varchar(255) DEFAULT NULL,
  `created_time` datetime DEFAULT NULL,
  `expected_duration` bigint(20) DEFAULT NULL,
  `expected_end` datetime DEFAULT NULL,
  `expected_start` datetime DEFAULT NULL,
  `job_data` varchar(255) DEFAULT NULL,
  `nominal_time` datetime DEFAULT NULL,
  `notification_msg` varchar(255) DEFAULT NULL,
  `parent_id` varchar(255) DEFAULT NULL,
  `sla_config` varchar(255) DEFAULT NULL,
  `upstream_apps` varchar(255) DEFAULT NULL,
  `user_name` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`job_id`),
  KEY `I_SL_RRTN_NOMINAL_TIME` (`nominal_time`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


DROP TABLE IF EXISTS `SLA_SUMMARY`;

CREATE TABLE `SLA_SUMMARY` (
  `job_id` varchar(255) NOT NULL,
  `actual_duration` bigint(20) DEFAULT NULL,
  `actual_end` datetime DEFAULT NULL,
  `actual_start` datetime DEFAULT NULL,
  `app_name` varchar(255) DEFAULT NULL,
  `app_type` varchar(255) DEFAULT NULL,
  `created_time` datetime DEFAULT NULL,
  `event_processed` tinyint(4) DEFAULT NULL,
  `event_status` varchar(255) DEFAULT NULL,
  `expected_duration` bigint(20) DEFAULT NULL,
  `expected_end` datetime DEFAULT NULL,
  `expected_start` datetime DEFAULT NULL,
  `job_status` varchar(255) DEFAULT NULL,
  `last_modified` datetime DEFAULT NULL,
  `nominal_time` datetime DEFAULT NULL,
  `parent_id` varchar(255) DEFAULT NULL,
  `sla_status` varchar(255) DEFAULT NULL,
  `user_name` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`job_id`),
  KEY `I_SL_SMRY_APP_NAME` (`app_name`),
  KEY `I_SL_SMRY_EVENT_PROCESSED` (`event_processed`),
  KEY `I_SL_SMRY_LAST_MODIFIED` (`last_modified`),
  KEY `I_SL_SMRY_NOMINAL_TIME` (`nominal_time`),
  KEY `I_SL_SMRY_PARENT_ID` (`parent_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


DROP TABLE IF EXISTS `VALIDATE_CONN`;

CREATE TABLE `VALIDATE_CONN` (
  `id` bigint(20) NOT NULL,
  `dummy` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


DROP TABLE IF EXISTS `WF_ACTIONS`;

CREATE TABLE `WF_ACTIONS` (
  `id` varchar(255) NOT NULL,
  `conf` mediumblob,
  `console_url` varchar(255) DEFAULT NULL,
  `created_time` datetime DEFAULT NULL,
  `cred` varchar(255) DEFAULT NULL,
  `data` mediumblob,
  `end_time` datetime DEFAULT NULL,
  `error_code` varchar(255) DEFAULT NULL,
  `error_message` varchar(500) DEFAULT NULL,
  `execution_path` varchar(1024) DEFAULT NULL,
  `external_child_ids` mediumblob,
  `external_id` varchar(255) DEFAULT NULL,
  `external_status` varchar(255) DEFAULT NULL,
  `last_check_time` datetime DEFAULT NULL,
  `log_token` varchar(255) DEFAULT NULL,
  `name` varchar(255) DEFAULT NULL,
  `pending` int(11) DEFAULT NULL,
  `pending_age` datetime DEFAULT NULL,
  `retries` int(11) DEFAULT NULL,
  `signal_value` varchar(255) DEFAULT NULL,
  `sla_xml` mediumblob,
  `start_time` datetime DEFAULT NULL,
  `stats` mediumblob,
  `status` varchar(255) DEFAULT NULL,
  `tracker_uri` varchar(255) DEFAULT NULL,
  `transition` varchar(255) DEFAULT NULL,
  `type` varchar(255) DEFAULT NULL,
  `user_retry_count` int(11) DEFAULT NULL,
  `user_retry_interval` int(11) DEFAULT NULL,
  `user_retry_max` int(11) DEFAULT NULL,
  `wf_id` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `I_WF_CTNS_PENDING_AGE` (`pending_age`),
  KEY `I_WF_CTNS_STATUS` (`status`),
  KEY `I_WF_CTNS_WF_ID` (`wf_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


DROP TABLE IF EXISTS `WF_JOBS`;

CREATE TABLE `WF_JOBS` (
  `id` varchar(255) NOT NULL,
  `app_name` varchar(255) DEFAULT NULL,
  `app_path` varchar(255) DEFAULT NULL,
  `conf` mediumblob,
  `created_time` datetime DEFAULT NULL,
  `end_time` datetime DEFAULT NULL,
  `external_id` varchar(255) DEFAULT NULL,
  `group_name` varchar(255) DEFAULT NULL,
  `last_modified_time` datetime DEFAULT NULL,
  `log_token` varchar(255) DEFAULT NULL,
  `parent_id` varchar(255) DEFAULT NULL,
  `proto_action_conf` mediumblob,
  `run` int(11) DEFAULT NULL,
  `sla_xml` mediumblob,
  `start_time` datetime DEFAULT NULL,
  `status` varchar(255) DEFAULT NULL,
  `user_name` varchar(255) DEFAULT NULL,
  `wf_instance` mediumblob,
  PRIMARY KEY (`id`),
  KEY `I_WF_JOBS_END_TIME` (`end_time`),
  KEY `I_WF_JOBS_EXTERNAL_ID` (`external_id`),
  KEY `I_WF_JOBS_LAST_MODIFIED_TIME` (`last_modified_time`),
  KEY `I_WF_JOBS_PARENT_ID` (`parent_id`),
  KEY `I_WF_JOBS_STATUS` (`status`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
2.hdfs 创建失败 确保 不存在原目录数据 （保护原数据）
3.IOException thrown while collecting data from host: No route to host
原因：agent开启了防火墙
解决：service iptables stop（不推荐）




4.CDH安装失败了，如何重新安装
1> 删除Agent节点的UUID 
      # rm -rf /opt/cloudera-manager/cm-5.7.0/lib/cloudera-scm-agent/*
2>  清空主节点CM数据库
      进入主节点的Mysql数据库，然后drop database cm;
3> 删除Agent节点namenode和datanode节点信息
     # rm -rf /dfs/nn/*
     # rm -rf /dfs/dn/*
4> 在主节点上重新初始化CM数据库
     # /opt/cloudera-manager/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p****** --scm-host localhost scm scm scm
5> 执行启动脚本
    主节点：# /opt/cloudera-manager/cm-5.7.0/etc/init.d/cloudera-scm-server start
     Agent节点：# /opt/cloudera-manager/cm-5.7.0/etc/init.d/cloudera-scm-agent start
过一会儿，即可通过主节点的7180端口重新进行CDH的安装 http://主节点IP:7180/cmf/login。
6.Yarn服务无法启动
    ①确保主机可以免密登陆其他机器 包括自己 
7.Host Monitor和Service Monitor
①ulimit -n 设置为 65535，.bashrc中添加了ulimit -HSn 65535
②删除/var/lib 下的Host Monitor和Service Monitor 文件夹，重启服务
8.所有主机
vim /etc/sysctl.conf
vm.swappiness=10
9.org.apache.hadoop.yarn.server.nodemanager.NodeManager
Error starting NodeManager
java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, 权限不够]
对执行目录授予运行用户的权限
10.hive metastore 启动不了情况下（此处可以进行对比）
停止hive服务，手动启动metastore服务 hive --service metastore，然后网页启动hive服务，成功！
注:此时数据库未使用MySQL,依然为内置数据库，修改hive-site.xml ：
关闭服务时手动关闭pid （ps -ef|grep metastore;kill -9 pid）
<property>
       <name>javax.jdo.option.ConnectionURL</name>
       <value>jdbc:mysql://192.168.1.181:3306/hive?characterEncoding=UTF-8</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>*****</value>
    </property>
驱动包建议5版本，然后依然手动开启后web开启
10.oozie 启动失败 可以选择跳过 直接进主页 然后手动开启oozie (手动建库建表)
11.Hive 建表 出现 “Specified key was too long; max key length is 767 bytes”
①Hive数据库编码问题，alter database hive character set latin1;
②更改表编码：
alter table PARTITIONS convert to character set latin1;
    alter table PARTITION_KEYS convert to character set latin1;
12.spark-shell 启动 权限问题
 su hdfs
 hdfs dfs -chmod 777 /user