ubuntu14使用hdp安装hadoop

1. 准备
1)安装java
copy jdk-7u80-linux-x64.tar.gz到本机,解压/test
vi /etc/profile
export JAVA_HOME=/test/jdk1.7.0_80
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
source /etc/profile
2)更改主机名
vi /etc/hostname
vi /etc/hosts
127.0.1.1       hdp01.test 改为173.168.0.121 hdp01.test 
173.168.0.121 hdp01.test
173.168.0.122 hdp02.test
3)sudo apt-get install ntp
service ntp start
4)安装postgres,mysql
第一步:安装 PostgreSQL, phpPgAdmin 和 Apache2
sudo apt-get -y install postgresql postgresql-contrib phppgadmin


第二步:配置 PostgreSQL 用户
su - postgres
psql
\password
\q


第3步:配置Postgresql
vim /etc/postgresql/9.1/main/postgresql.conf


 
Ubuntu上安装MySQL非常简单只需要几条命令就可以完成。
sudo apt-get install mysql-server
sudo apt-get install mysql-client libmysql-java
sudo apt-get install libmysqlclient-dev
sudo apt-get install  libmysql-java


2. 切换到root,设置ssh
1) 安装ssh
dpkg --list | grep ssh
sudo apt-get install openssh-server
2) 生成rsa密钥
ssh-keygen -t rsa -P ''
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
3)更改root、ssh及文件权限
chmod o+w ~/ 
chmod 700 -R ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod 755 /root
4)更改sshd_config
vi /etc/ssh/sshd_config
将/etc/ssh/sshd_config文件中的  PermitRootLogin prohibit-password改为PermitRootLogin yes(或者删掉PermitRootLogin prohibit-password也可以,为了安全可以备份,注释掉就可以)
sudo /etc/init.d/ssh restart
5)
vi ~/.ssh/config
host 173.168.0.121
user [YOUR_USER_NAME]
6)在slave机器上:
在root目录下
mkdir .ssh
复制master上的id_rsa.pub到.ssh
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod o+w ~/ 
chmod 700 -R ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod 755 /root


3.从以下三个位置下载包,可以直接在windows上用http下载
nohup wget -c http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.1.0/ambari-2.2.1.0-ubuntu14.tar.gz > 1.log 2>&1 &
nohup wget -c http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.4.0.0/HDP-2.4.0.0-ubuntu14-deb.tar.gz > 1.log 2>&1 &
nohup wget -c http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/ubuntu14/HDP-UTILS-1.1.0.20-ubuntu14.tar.gz > 2.log 2>&1 &
1) copy这三个文件到/var/www/html/hadoop
cd /var/www/html
mkdir hadoop
2) 运行http服务,解压三个文件到/var/www/html/hadoop


4. 在内网其它机器,添加官方源
sudo wget -nv http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.1.0/ambari.list -O /etc/apt/sources.list.d/ambari.list
sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD
5. 修改官方源配置文件(dist上级目录)
sudo vi /etc/apt/sources.list.d/ambari.list


#VERSION_NUMBER=2.2.1.0-161
#deb http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.1.0 Ambari main


deb http://173.168.0.121/hadoop/AMBARI-2.2.1.0/ubuntu14/2.2.1.0-161/ Ambari main


6. 更新源并安装程序
sudo apt-get update
master:sudo apt-get install ambari-server
slave:  sudo apt-get install ambari-agent


7. 配置ambari服务
1)sudo ambari-server setup
输入JAVA_HOME的值,其他默认
2)ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql.jar


8. 启动服务
master:sudo ambari-server start
slave:  sudo ambari-agent start
sudo vi /etc/ambari-agent/conf/ambari-agent.ini 
hostname=hdp01.test


9. 配置hive,ambari数据库
1)
CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost';
CREATE USER 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%';
CREATE USER 'hive'@'hdp01.test'IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'hdp01.test';
FLUSH PRIVILEGES;
create DATABASE hive;


CREATE USER 'ambari'@'%' IDENTIFIED BY 'bigdata'; 
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'bigdata';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost';
CREATE DATABASE ambari;
USE ambari;
SOURCE /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;


CREATE USER 'oozie'@'localhost' IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'localhost';
CREATE USER 'oozie'@'%' IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%';
CREATE USER 'oozie'@'hdp01.test'IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'hdp01.test';
FLUSH PRIVILEGES;
create DATABASE oozie;


2)
sudo vi /etc/mysql/my.cnf 
添加'#'注释掉其中的"bind-address = 127.0.0.1"
sudo service mysql restart


10. 访问web界面
http://ip:8080
使用默认的admin/admin登录。
1)点击左上角图标或者选择 Launch Install Wizard
2)第二步Select Stack,在选择源的地方,填写路径(dist上级目录)
http://173.168.0.121/hadoop/HDP/ubuntu14/2.x/updates/2.4.0.0/
http://173.168.0.121/hadoop/HDP-UTILS-1.1.0.20/repos/ubuntu14/


11. 在第3步


1)只填hdp01.test
sudo cat id_rsa
把private key复制到框里,点击下一步,看到sucess后,点击Back
2)把hdp02.test也填上
选择手动安装ambari-agent
看到两个都sucess后,点击下一步


12. 禁止HDP(Transparent HugePages)
vi /etc/default/grub,在GRUB_CMDLINE_LINUX加入选项 transparent_hugepage=never
grub-mkconfig -o /boot/grub/grub.cfg

重启


13. hive、oozie选择已有的mysql数据库,设置数据库密码

 




阅读更多
相关热词
换一批

没有更多推荐了,返回首页