1. 安装前的准备
准备好Ubuntu离线source,配置/etc/apt/source.list 准备好Cloudera Hadoop离线source,配置/etc/apt/source.list.d/cloudera.list
同步集群的/etc/hosts
ssh免密码登录,cloudera-scm-server所在机器能够免密码登录其他cloudera-scm-agent机器或者所有机器的root用户密码相同
2. 系统环境
OS:ubuntu12.04CDH:cdh3u6
Cloudera manager:4.8.2
3. 数据库的配置
3.1 安装配置MySQL
Cloudera Manager支持三种数据库类型:MySQL、PostgreSQL、Oracle。我们选择使用MySQL。apt-get install mysql-server
service mysql start
在/etc/mysql/conf.d下创建mysql_cloudera_manager.cnf文件,加入以下内容:
[mysqld]
transaction-isolation=READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links=0
key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 16M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
# Important: see Configuring the Databases and Setting max_connections
max_connections = 550
# log-bin should be on a disk with enough free space
log-bin=/x/home/mysql/logs/binary/mysql_binary_log
# For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
3.2 配置innodb
mv /var/lib/mysql/ib_logfile* ~重启mysql
service mysql restart
3.3 安装mysql-connector
apt-get install libmysql-java3.4 创建数据库
需要创建的数据库列表:amon smon ran hmon hive
mysql -u root -p iflytek
create database amon DEFAULT CHARACTER SET utf8;
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';
create database smon DEFAULT CHARACTER SET utf8;
grant all on smon.* TO 'smon'@'%' IDENTIFIED BY 'smon_password';
create database rman DEFAULT CHARACTER SET utf8;
grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';
create database hmon DEFAULT CHARACTER SET utf8;
grant all on hmon.* TO 'hmon'@'%' IDENTIFIED BY 'hmon_password';
create database hive DEFAULT CHARACTER SET utf8;
grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';
备份数据库
mysqldump -piflytek amon > /tmp/amon-backup.sql
和其他集群数据库
3.5 安装Cloudera Manager
添加Cloudera Manager的源创建/etc/apt/source.list.d/cloudera-manager.list文件,并在文件中添加
deb [arch=amd64] http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh precise-cdh4 contrib
deb-src http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh precise-cdh4 contrib
deb http://archive.cloudera.com/cm4/ubuntu/precise/amd64/cm precise-cm4 contrib
deb-src http://archive.cloudera.com/cm4/ubuntu/precise/amd64/cm precise-cm4 contrib
保存后下载key
curl -s http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh/archive.key | sudo apt-key add -
若上述命令执行失败,请自己下载key文件导入。apt-key add [file path]
在Server机器上执行
apt-get install cloudera-manager-daemons cloudera-manager-server
在agent机器上执行
apt-get install cloudera-manager-agent cloudera-manager-daemons
3.6 为cloudera manager准备数据库
执行/usr/share/cmf/schema/scm_prepare_database.sh database-type [options] database-name username password如
./scm_prepare_database.sh mysql amon root xxx
./scm_prepare_database.sh mysql smon root xxx
./scm_prepare_database.sh mysql hmon root xxx
./scm_prepare_database.sh mysql rman root xxx
./scm_prepare_database.sh mysql hive root xxx
3.7 安装HDFS
指定一台机器做namendoeapt-get install namenode
在指定的DataNode机器上执行
apt-get install datanode
3.8 启动cloudera manager Server和agent
在所有机器上创建一些保存日志的目录:mkdir -p /var/log/cloudera-scm-headlamp
mkdir -p /var/log/cloudera-scm-firehose
mkdir -p /var/log/cloudera-scm-alertpublisher
mkdir -p /var/log/cloudera-scm-eventserver
mkdir -p /var/lib/cloudera-scm-headlamp
mkdir -p /var/lib/cloudera-scm-firehose
mkdir -p /var/lib/cloudera-scm-alertpublisher
mkdir -p /var/lib/cloudera-scm-eventserver
创建完成后修改owner
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-*
chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-*
如果使用tar包安装需要chown到cloudera-scm用户组下。如果使用apt-get方式安装不需要chown。默认使用的是root。
修改所有agent的JDK配置
在/etc/default/ cloudera-scm-agent中增加
export CMF_AGENT_JAVA_HOME=/usr/local/jdk6
在Server机器上执行:
service cloudera-scm-server start
分别登录agent机器执行:
service cloudera-scm-agent start
3.9 登录web端
登录部署有scm-server机器的http://ServerIP:7180用户名:admin
密码:admin
3.10 数据库设置
数据库主机名称:localhost,用户名:root,数据库名称上面那5个,密码:xxxx,即MySQL数据库的密码。4. web端使用
apt-get install oracle-j2sdk1.6
附上另一种离线安装方法:
官网安装用户手册
ubuntu系列:
http://archive.cloudera.com/cm4/cm/4/
http://archive.cloudera.com/search/parcels/latest/
http://archive.cloudera.com/impala/parcels/latest/
http://archive.cloudera.com/cdh4/parcels/latest/
http://archive.cloudera.com/cm4/ubuntu/precise/amd64/cm/pool/contrib/o/oracle-j2sdk1.6/
http://archive.cloudera.com/cm4/cm/4/
http://archive.cloudera.com/search/parcels/latest/
http://archive.cloudera.com/impala/parcels/latest/
http://archive.cloudera.com/cdh4/parcels/latest/
http://archive.cloudera.com/cm4/ubuntu/precise/amd64/cm/pool/contrib/o/oracle-j2sdk1.6/