以三个节点为例,搭建cloudera集群。
总体思路是,现在master1上安装好cloudera,然后在cloudera manager上通过添加hosts的方式加入其他两个节点。
1./etc/hosts设置
按master1,slave1,slave2给三台机器配置hostname
修改所有节点的/etc/hosts
192.168.23.131 master1 192.168.23.130 slave1 192.168.23.132 slave2
确保所有机器root密码一致或者sudo免输入密码用户的用户名、密码一致!
2.添加Cloudera源 1.添加源
sudo nano /etc/apt/sources.list.d/cloudera.list
写入
# Packages for Cloudera Manager, Version 5, on Ubuntu 14.04 x86_64 deb [arch=amd64] http://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm trusty-cm5 contrib deb-src http://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm trusty-cm5 contrib # Packages for Cloudera's Distribution for Hadoop, Version 5, on Ubuntu 14.04 amd64 deb [arch=amd64] http://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh trusty-cdh5 contrib deb-src http://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh trusty-cdh5 contrib
http://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/ 为cm的源
http://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh trusty-cdh5 为chd5的源
上述源都是对应的ubuntu14的源,如果需要找到ubuntu10和ubuntu12的源,请在
对应目录寻找cloudera.list
lucid -> ubuntu10 precise -> ubuntu12 trusty -> ubuntu14
获取apt key
curl -s http://http://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh/archive.key| sudo apt-key add -
刷新源列表
3.安装java环境
sudo apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install oracle-j2sdk1.7
java的安装路径参考:
/usr/lib/jvm
4.安装mysql以及JDBC驱动
sudo apt-get install mysql-server libmysql-java
5.配置mysql
sudo nano /etc/mysql/conf.d/mysql_cloudera_manager.cnf
复制代码
写入
[mysqld] transaction-isolation=READ-COMMITTED # Disabling symbolic-links is recommended to prevent assorted security risks; # to do so, uncomment this line: # symbolic-links=0 key_buffer = 16M key_buffer_size = 32M max_allowed_packet = 16M thread_stack = 256K thread_cache_size = 64 query_cache_limit = 8M query_cache_size = 64M query_cache_type = 1 # Important: see Configuring the Databases and Setting max_connections max_connections = 550 # log-bin should be on a disk with enough free space log-bin=/var/log/mysql/mysql_binary_log # For MySQL version 5.1.8 or later. Comment out binlog_format for older versions. binlog_format = mixed read_buffer_size = 2M read_rnd_buffer_size = 16M sort_buffer_size = 8M join_buffer_size = 8M # InnoDB settings innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 64M innodb_buffer_pool_size = 4G innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid
复制代码
在安装的过程中一定保证内存足够大,否则会遇到下面问题
上面配置需要根据自己的实际情况,在配置过程中重启mysql的时候,发生了下面错误
stop: Unknown instance: start: Job failed to start
注意:不能够重启是因为这两项的原因
innodb_buffer_pool_size = 4G
log-bin=/var/log/mysql/mysql_binary_log
编辑my.cnf
sudo nano /etc/mysql/my.cnf
把下面这一行注释掉
#bind-address = 127.0.0.1
6.配置innodb
sudo su mv /var/lib/mysql/ib_logfile* /var/tmp/ logout
7.初始化数据库
sudo service mysql restart
create database amon DEFAULT CHARACTER SET utf8; grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password'; grant all on amon.* TO 'amon'@'CDH' IDENTIFIED BY 'amon_password'; create database smon DEFAULT CHARACTER SET utf8; grant all on smon.* TO 'smon'@'%' IDENTIFIED BY 'smon_password'; grant all on smon.* TO 'smon'@'CDH' IDENTIFIED BY 'smon_password'; create database rman DEFAULT CHARACTER SET utf8; grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password'; grant all on rman.* TO 'rman'@'CDH' IDENTIFIED BY 'rman_password'; create database hmon DEFAULT CHARACTER SET utf8; grant all on hmon.* TO 'hmon'@'%' IDENTIFIED BY 'hmon_password'; grant all on hmon.* TO 'hmon'@'CDH' IDENTIFIED BY 'hmon_password'; create database hive DEFAULT CHARACTER SET utf8; grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive_password'; grant all on hive.* TO 'hive'@'CDH' IDENTIFIED BY 'hive_password';
8.安装 cloudera-manager以及agent(因为master也是一个节点)
sudo apt-get install cloudera-manager-daemons cloudera-manager-server cloudera-manager-agent
9.配置cloudera-manager-server数据库
sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p --scm-host localhost scm scm scm_password
这里执行的时候,如果设置了mysql的密码,那么需要输入mysql密码,并且会出现log4j错误,不用担心。