保证mysql在server端安装成功,查看集群配置,其他库的使用情况是否一致,一般情况下仅仅修正scm库。查看psql中数据
mysql的my.cnf配置修改
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Recommended in standard MySQL setup
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
server-id=1
# For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
mysql创建数据库(主要对scm数据库进行迁移,可选可不选)
删除database:
DROP DATABASE scm;
DROP DATABASE amon;
DROP DATABASE rman;
DROP DATABASE metastore;
DROP DATABASE sentry;
DROP DATABASE nav;
DROP DATABASE navms;
创建database:
create database amon DEFAULT CHARACTER SET utf8;
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';
create database rman DEFAULT CHARACTER SET utf8;
grant all on rman.* TO'rman'@'%' IDENTIFIED BY 'rman_password';
create database metastore DEFAULT CHARACTER SET utf8;
grant all on metastore.*TO 'hive'@'%' IDENTIFIED BY 'hive_password';
create database sentry DEFAULT CHARACTER SET utf8;
grant all on sentry.* TO'sentry'@'%' IDENTIFIED BY 'sentry_password';
create database nav DEFAULT CHARACTER SET utf8;
grant all on nav.* TO 'nav'@'%'IDENTIFIED BY 'nav_password';
create database navms DEFAULT CHARACTER SET utf8;
grant all on navms.* TO 'navms'@'%'IDENTIFIED BY 'navms_password';
mysql保证innodb引擎
show global variables like "have_%";
show plugins;
show variables like "have_dynamic%"; 是否支持动态加载插件
show variables like 'plugin_dir'; #找到mysql存放插件的目录
INSTALL PLUGIN InnoDB SONAME 'ha_innodb.so';
http://www.linuxidc.com/Linux/2015-04/116167.htm
1.停止服务
1.停止集群
2.停止cm的service
3.停止bn00-bn04的agent
sudo service cloudera-scm-agent stop
2.备份cm的api
mkdir -p /home/user/deployment
curl -v -u admin:admin http://bn00:7180/api/v14/cm/deployment > /home/user/deployment
3.对postgre的db和server目录做备份
cd /var/lib/cloudera-scm-server-db/
tar -cvf /home/user/cloudera-scm-server-db.tar .
cd /etc/cloudera-scm-server/
tar -cvf /home/user/cloudera-scm-server.tar .
4.在bn00端停止server,db
sudo service cloudera-scm-server stop
sudo service cloudera-scm-server-db stop
sudo chkconfig cloudera-scm-server-db off
5.mysql中创建temp
#mysql -h mysql_hostname -u user -p
mysql> create database temp;
mysql> grant all on *.* to 'temp'@'%' identified by 'temp' with grant option;
#/usr/share/cmf/schema/scm_prepare_database.sh mysql -h bn00 -utemp -ptemp --scm-host bn00 scm scm scm #创建了scm
显示信息最后一行如下:
All done, your Cloudera Manager database is ready to go!
则代表执行成功。删除temp user
mysql>drop user 'temp'@'%';
6.在bn00端开启server
sudo service httpd start
sudo service cloudera-scm-server start
7.登录UI界面,恢复API,恢复集群
$ curl --upload-file /home/user/deployment -u admin:admin http://bn00:7180/api/v14/cm/deployment?deleteCurrentDeployment=true -H "Content-Type: application/json"
8.刷新UI界面,目前所有服务都在,处于待重启状态。
9.删除旧的guid(所有主机)(主机状况运行不良情况)
rm -rf /var/lib/cloudera-scm-agent/cm_guid
10.主机-所有主机-重新运行主机升级向导,注意使用密钥。
11.查看parcel状态是否正常,进行激活操作。
可能的问题:
parcel处没有任何操作可以进行,卡住了。
解决方案:
http://node0:7180/static/apidocs/ API文档
1.在http://bn00:7180/api/v14/cm/deployment中找到对应parcel相应的配置以及版本。
2.curl -u admin:admin -X POST http://node0:7180/api/v14/clusters/cluster/parcels/products/CDH/versions/5.7.5-1.cdh5.7.5.p0.3/commands/deactivate
3.curl -u admin:admin -X POST http://bn00:7180/api/v14/clusters/cluster/parcels/products/KAFKA/versions/2.0.1-1.2.0.1.p0.5/commands/deactivate
postgresql相关密码查询
http://blog.csdn.net/yujin2010good/article/details/72482007
bn00数据库相关配置
Hbase
Hive
Hue
amon
rman
metastore
sentry
nav
navms
(在采用Cloudera-Manager安装cdh时,通常使用内嵌的PostgreSQL数据库。 Cloudera-Manager除了保存CDH集群的配置元数据的scm数据库外,还为Activity Monitor(活动监控)、Service Monitor(服务监控)、Report Manager(报告管理)、Host Monitor(主机监控)、Cloudera Navigator(Cloudera导航)等信息分别创建数据amon、smon、rmon、hmon、nav相应的数据。)
http://blog.csdn.net/yujin2010good/article/details/72482007
http://www.cnblogs.com/jxrichar/p/7240812.html
psql的基本命令
>
一、建立数据库连接
接入PostgreSQL数据库: psql -h IP地址 -p 端口 -U 数据库名
>
之后会要求输入数据库密码
>
二、访问数据库
>
1、列举数据库:\l
>
2、选择数据库:\c 数据库名
3、查看该某个库中的所有表:\dt
4、切换数据库:\c interface
5、查看某个库中的某个表结构:\d 表名
6、查看某个库中某个表的记录:select * from apps limit 1;
7、显示字符集:\encoding
8、退出psgl:\q
各种坑
>
1.代理节点启动报错(主机状况不良):
错误信息:Error, CM server guid updated, expected df16790a-2e44-44ec-9db2-8731cc635c61, received b6fecabc-8e32-46be-8a43-5f261064b2c7
解决方法:删除/var/lib/cloudera-scm-agent下cm_guid文件
>
2.oozie缺少ext-2.2:
解决方法:将ext-2.2拷贝至/var/lib/oozie目录
>
3.spark运行报错:Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster!
Please check the values of ‘yarn.scheduler.maximum-allocation-mb’ and/or ‘yarn.nodemanager.resource.memory-mb’
解决方法:修改服务YARN (MR2 Included)中如下配置项并保存,重启YARN:
最小容器内存(大于1G):yarn.scheduler.minimum-allocation-mb
最大容器内存(大于1G):yarn.scheduler.maximum-allocation-mb
容器内存(大于1G):yarn.nodemanager.resource.memory-mb
>
4.spark运行报错:org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode=”/user”:hdfs:supergroup:drwxr-xr-x
解决方法:将export HADOOP_USER_NAME=hdfs添加到~/.bash_profile中,然后source ~/.bash_profile
>
5.hive执行引擎默认为mapreduce的配置修改:hive.execution.engine由mr改为spark
>
6.