hadoop/hbase 环境搭建

最新推荐文章于 2023-06-28 22:16:12 发布

木木周

最新推荐文章于 2023-06-28 22:16:12 发布

阅读量1.2k

点赞数

本文链接：https://blog.csdn.net/u011779268/article/details/75578888

版权

本文详细介绍了如何在多台机器上搭建Hadoop和HBase集群。包括创建用户组和用户，配置SSH信任，安装软件，修改配置文件如hadoop-env.sh、core-site.xml、hdfs-site.xml等，格式化HDFS和ZK，启动Hadoop的DFS和YARN，以及安装和配置HBase集群。最后提到了解决权限问题和伪分布式的部署步骤。

摘要由CSDN通过智能技术生成

hadoop安装

（1）创建hadoop用户组和用户见（1.1，1.5）

（2）配置hadoop用户信任关系

1）生成非对称密钥：ssh-keygen -t rsa // 一直回车

1）znode01,znode02,znode03 添加信任关系

(1)znode01 无密码访问znode02

(2)在znode02 ：mkdir ~/.ssh/tmp ; touch ~/.ssh/authorized_keys;

(3)在znode01 ：scp ~/.ssh/id_rsa.pub hdpusr@znode02:~/.ssh/tmp //此时输入znode02密码即可

(4)在znode02 ： cat ~/.ssh/tmp/id_rsa.pub >> ~/.ssh/authorized_keys

(5)znode01 ssh 无密码访问自己 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

......重复以上相似操作配置其他机器

（3）安装包hadoop-2.6.0.zip准备

mkdir ~/env;mkdir ~/env/hadoop；将hadoop-2.6.0.zip 解压到hadoop

（4）配置文件修改

（1）修改hadoop-env.sh

配置JAVA_HOME: export JAVA_HOME=/usr/software/jdk1.7.0_79

（2）修改core-site.xml

<name>fs.defaultFS</name>

<value>hdfs://hdpcls1</value>

</property>

<name>io.file.buffer.size</name>

</property>

<name>hadoop.tmp.dir</name>

<value>/home/hdpusr/env/hadoop/tmp</value>

</property>

<name>ha.zookeeper.quorum</name>

<value>znode01:29181,znode02:29181,znode03:29181</value>

</property>

</configuration>

（3）修改hdfs-stie.xml

<name>dfs.nameservices</name>

<value>hdpcls1</value>

</property>

<name>dfs.ha.namenodes.hdpcls1</name>

</property>

<name>dfs.namenode.rpc-address.hdpcls1.nn1</name>

<value>znode01:8920</value>

</property>

<name>dfs.namenode.http-address.hdpcls1.nn1</name>

<value>znode01:59070</value>

</property>

<name>dfs.namenode.rpc-address.hdpcls1.nn2</name>

<value>znode02:8920</value>

</property>

<name>dfs.namenode.http-address.hdpcls1.nn2</name>

<value>znode02:59070</value>

</property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://znode01:8485;znode02:8485;znode03:8485/hdpcls1</value>

</property>

<name>dfs.journalnode.edits.dir</name>

<value>/home/hdpusr/env/hadoop/dfs/jn</value>

</property>

<name>dfs.journalnode.rpc-address</name>

</property>

<name>dfs.journalnode.http-address</name>

</property>

<name>dfs.journalnode.https-address</name>

</property>

<name>dfs.ha.automatic-failover.enabled</name>

</property>

<name>dfs.client.failover.proxy.provider.hdpcls1</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

</property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/home/hdpusr/.ssh/id_rsa</value>

</property>

<name>dfs.ha.fencing.ssh.connect-timeout</name>

</property>

<name>dfs.datanode.data.dir</name>

<value>file:/home/hdpusr/env/hadoop/dfs/dn/data01</value>

</property>

</configuration>

（4）修改mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

</configuration>

（5）修改yarn-site.xml

<name>yarn.resourcemanager.ha.enabled</name>

</property>

<name>yarn.resourcemanager.cluster-id</name>

<value>rmcls1</value>

</property>

<name>yarn.resourcemanager.ha.rm-ids</name>

</property>

<name>yarn.resourcemanager.hostname.rm1</name>

<value>znode01</value>

</property>

<name>yarn.resourcemanager.hostname.rm2</name>

<value>znode02</value>

</property>

<name>yarn.resourcemanager.recovery.enabled</name>

</property>

<name>yarn.resourcemanager.store.class</name>

<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>

</property>

<name>yarn.resourcemanager.zk-address</name>

<value>znode01:29181,znode02:29181,znode03:29181</value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

（6）修改slaves

vim slaves

添加

znode02

znode03

（7）创建hadoop目录

mkdir -p ~/env/hadoop/{tmp,dfs/jn,dfs/dn/data01}

（5）环境变量配置

su - hdpusr01

vim ~/.bash_profile

export HADOOP_HOME=${HOME}/env/hadoop

export HADOOP_CONF_DIR=$HOME/env/hadoop/etc/hadoop

#export HADOOP_PID_DIR=$HOME/env/hadoop/pid

export HADOOP_LOG_DIR=$HOME/env/hadoop/logs

export HADOOP_LIBEXEC_DIR=$HOME/env/hadoop/libexec

export HADOOP_PREFIX=${HADOOP_HOME}

export HADOOP_COMMON_HOME=${HADOOP_HOME}

export HADOOP_HDFS_HOME=${HADOOP_HOME}

export HADOOP_LIB=${HADOOP_HOME}/lib

export HADOOP_LIBRARY_PATH=${HADOOP_HOME}/lib/native

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export HADOOP_MAPRED_HOME=${HADOOP_HOME}

export HADOOP_YARN_USER=pochdp04

export YARN_CONF_DIR=${HADOOP_CONF_DIR}

export YARN_HOME=${HADOOP_HOME}

export HADOOP_YARN_HOME=${HADOOP_HOME}

export YARN_LOG_DIR=${HADOOP_LOG_DIR}

export YARN_PID_DIR=${HADOOP_CONF_DIR}/../yarn

source ~/.bash_profile 使环境变量生效

（6）格式化操作，启动停止操作

(1) 保证zk正常运行

(2)znode01,znode02,znode03 启动sbin目录下执行：./hadoop-daemon.sh start journalnode

28365 JournalNode --正常的情况可以看到该进程

（3）格式化hdfs

znode01 节点bin目录执行：./hdfs namenode -format

显示：

15/12/18 13:34:35 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1518459121-10.46.52.30-1450416875417

15/12/18 13:34:35 INFO common.Storage: Storage directory /aifs01/users/hdpusr01/hadoop/tmp/dfs/name has been successfully formatted.

15/12/18 13:34:35 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

15/12/18 13:34:35 INFO util.ExitUtil: Exiting with status 0

【将/home/hdpusr/env/hadoop/tmp 拷贝之znode02 /home/hdpusr/env/hadoop/tmp 下】

（4）格式化ZK

znode01 节点bin目录执行：./hdfs zkfc -formatZK

显示：

15/12/18 13:49:04 INFO ha.ActiveStandbyElector: Session connected.

15/12/18 13:49:04 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/hdpcls1 in ZK.

15/12/18 13:49:04 INFO zookeeper.ZooKeeper: Session: 0x251b09471e80001 closed

15/12/18 13:49:04 INFO zookeeper.ClientCnxn: EventThread shut down

成功后zk 出现节点 hadoop-ha

[zk: 127.0.0.1:29181(CONNECTED) 1] ls /

[consumers, config, hadoop-ha, zookeeper, brokers, admin, controller_epoch]

（5）启动分布式存储 znode01 sbin上执行： ./start-dfs.sh

启动成功

znode01 进程：

15507 NameNode

14371 JournalNode

23478 DFSZKFailoverController

znode02 进程：

23870 DataNode

23474 JournalNode

23998 DFSZKFailoverController

znode03 进程

16268 JournalNode

16550 DataNode

同时zk注册节点

[zk: 127.0.0.1:29181(CONNECTED) 4] ls /hadoop-ha/hdpcls1

[ActiveBreadCrumb, ActiveStandbyElectorLock]

此时可在页面输入http://118.190.79.15:59070 访问

（6）启动分布式资源管理

znode01 sbin上执行：./start-yarn.sh

此时znode01,

出现进程：17746 ResourceManager

znode02

出现进程：24975 NodeManager

znode03

出现进程17389 NodeManager

Zk注册节点：rmstore, yarn-leader-election

[zk: 127.0.0.1:29181(CONNECTED) 0] ls /

[hadoop-ha, admin, zookeeper, consumers, config, rmstore, yarn-leader-election, brokers, controller_epoch]

(7) 全部启动：./start-all.sh，全部停止./stop-all.sh

[hdpusr@znode01 sbin]$ ./stop-all.sh

This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh

17/03/25 09:56:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Stopping namenodes on [znode01 znode02]

znode02: no namenode to stop

znode01: stopping namenode

znode03: stopping datanode

znode02: stopping datanode

Stopping journal nodes [znode01 znode02 znode03]

znode03: stopping journalnode

znode02: stopping journalnode

znode01: stopping journalnode

17/03/25 09:57:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Stopping ZK Failover Controllers on NN hosts [znode01 znode02]

znode02: stopping zkfc

znode01: no zkfc to stop

stopping yarn daemons

stopping resourcemanager

znode02: stopping nodemanager

znode03: stopping nodemanager

no proxyserver to stop

[hdpusr@znode01 sbin]$ jps

22992 Jps

[hdpusr@znode01 sbin]$ start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

17/03/25 09:57:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on [znode01 znode02]

znode02: starting namenode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-namenode-znode02.out

znode01: starting namenode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-namenode-znode01.out

znode03: starting datanode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-datanode-znode03.out

znode02: starting datanode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-datanode-znode02.out

Starting journal nodes [znode01 znode02 znode03]

znode02: starting journalnode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-journalnode-znode02.out

znode03: starting journalnode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-journalnode-znode03.out

znode01: starting journalnode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-journalnode-znode01.out

17/03/25 09:57:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting ZK Failover Controllers on NN hosts [znode01 znode02]

znode02: starting zkfc, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-zkfc-znode02.out

znode01: starting zkfc, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-zkfc-znode01.out

starting yarn daemons

starting resourcemanager, logging to /home/hdpusr/env/hadoop/logs/yarn-hdpusr-resourcemanager-znode01.out

znode03: starting nodemanager, logging to /home/hdpusr/env/hadoop/logs/yarn-hdpusr-nodemanager-znode03.out

znode02: starting nodemanager, logging to /home/hdpusr/env/hadoop/logs/yarn-hdpusr-nodemanager-znode02.out

（8）查看集群状态 bin目录 ./hdfs dfsadmin -report

（9）http://118.190.79.15:59070/dfshealth.html#tab-overview

***********************************************hadoop完全分布式部署**********************************************
0.集群规划
主机名 ip地址安装的软件运行的进程
HBS01 10.46.52.30 hadoop,hbase namenode,zkfc,resourcemanager
HBS02 10.46.52.31 hadoop namenode,zkfc,resourcemanager
HBS03 10.46.52.32 hadoop,hbase datanode
HBS04 10.46.52.33 hadoop,zookeeper,hbase datanode,nodemanager,journalnode
HBS05 10.46.52.34 hadoop,zookeeper,hbase datanode,nodemanager,journalnode
HBS06 10.46.52.35 hadoop,zookeeper,hbase datanode,nodemanager,journalnode

1.创建用户以及配置主机名
mkdir -p /zn/users
useradd -u 351 -g hadoop -G ibss -d /zn/users/hdpusr01 -m hdpusr01
passwd hdpusr01 --stdin <<< Linuhdp_0805
echo -e "\n. ~puwadm/wprofile\n" >> ~hdpusr01/.bash_profile

useradd -u 352 -g hbase -G ibss -d /zn/users/hbsusr01 -m hbsusr01
passwd hbsusr01 --stdin <<< Linuhbs_0805
echo -e "\n. ~puwadm/wprofile\n" >> ~hbsusr01/.bash_profile

vi /etc/hosts
10.46.52.30 HBS01
10.46.52.31 HBS02
10.46.52.32 HBS03
10.46.52.33 HBS04
10.46.52.34 HBS05
10.46.52.35 HBS06

2.配置互信
--每台主机都需要操作
su - hdpusr01
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
chmod 600 .ssh/authorized_keys

--选择一台master操作(HBS01)
ssh HBS02 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS03 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS04 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS05 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS06 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

scp ~/.ssh/authorized_keys HBS02:/zn/users/hdpusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS03:/zn/users/hdpusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS04:/zn/users/hdpusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS05:/zn/users/hdpusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS06:/zn/users/hdpusr01/.ssh/authorized_keys

--验证配置是否生效
ssh HBS01 date
ssh HBS02 date
ssh HBS03 date
ssh HBS04 date
ssh HBS05 date
ssh HBS06 date

3.安装jdk
which java

4.安装Hadoop
4.1 安装zookkeper集群(HBS04 - HBS06)
--配置zookeeper集群
[root@HBS04 hdpusr01]# tar xvf /opt/software/zookeeper.ccs01.tgz
[root@HBS04 hdpusr01]# chown -R hdpusr01:hadoop zookeeper/
[root@HBS04 hdpusr01]# rm -rf zookeeper/logs/*
[root@HBS04 hdpusr01]# rm -rf zookeeper/zk-data/version-2/
[root@HBS04 hdpusr01]# rm -rf zookeeper/zk-data/zookeeper_server.pid
[root@HBS04 hdpusr01]# vi zookeeper/conf/zoo.cfg
dataDir=/zn/users/hdpusr01/zookeeper/zk-data
server.1=HBS04:29888:39889
server.2=HBS05:29888:39889
server.3=HBS06:29888:39889

hdpusr01@HBS04:/zn/users/hdpusr01> vi zookeeper/bin/zkEnv.sh
export ZOOCFGDIR=/zn/users/hdpusr01/zookeeper/conf
export ZOO_LOG_DIR=/zn/users/hdpusr01/zookeeper/logs

[root@HBS04 hdpusr01]# su - hdpusr01
hdpusr01@HBS04:/zn/users/hdpusr01> tar cvf zkper.tar zookeeper/
hdpusr01@HBS04:/zn/users/hdpusr01> scp zkper.tar HBS05:/zn/users/hdpusr01
hdpusr01@HBS04:/zn/users/hdpusr01> scp zkper.tar HBS06:/zn/users/hdpusr01

hdpusr01@HBS05:/zn/users/hdpusr01> tar xvf zkper.tar
hdpusr01@HBS05:/zn/users/hdpusr01> echo 2 >zookeeper/zk-data/myid

hdpusr01@HBS06:/zn/users/hdpusr01> tar xvf zkper.tar
hdpusr01@HBS06:/zn/users/hdpusr01> echo 3 >zookeeper/zk-data/myid

--启动zookeeper
hdpusr01@HBS04:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh start
hdpusr01@HBS05:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh start
hdpusr01@HBS06:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh start

4.2 安装hadoop集群（AI-OPT-HBS01 - HBS06）
[root@HBS01 hdpusr01]# tar xvf /opt/software/hadoop-2.6.0.tgz
[root@HBS01 hdpusr01]# chown -R hdpusr01:hadoop hadoop-2.6.0/
hdpusr01@HBS01:/zn/users/hdpusr01> mv hadoop-2.6.0 hadoop

--修改hadoop-env.sh
hdpusr01@HBS01:/zn/users/hdpusr01> vi hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/opt/freeware/jdk1.7.0_79

--修改core-site.xml
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop> vi core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hdpcls1</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/zn/users/hdpusr01/hadoop/tmp</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>HBS04:29181,HBS05:29181,HBS06:29181</value>
</property>
</configuration>

--修改hdfs-stie.xml
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop> vi hdfs-site.xml
<configuration>

<property>
<name>dfs.nameservices</name>
<value>hdpcls1</value>
</property>

<property>
<name>dfs.ha.namenodes.hdpcls1</name>
<value>nn1,nn2</value>
</property>

<property>
<name>dfs.namenode.rpc-address.hdpcls1.nn1</name>
<value>HBS01:8920</value>
</property>

<property>
<name>dfs.namenode.http-address.hdpcls1.nn1</name>
<value>HBS01:59070</value>
</property>

<property>
<name>dfs.namenode.rpc-address.hdpcls1.nn2</name>
<value>HBS02:8920</value>
</property>

<property>
<name>dfs.namenode.http-address.hdpcls1.nn2</name>
<value>HBS02:59070</value>
</property>

<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://HBS04:8485;HBS05:8485;HBS06:8485/hdpcls1</value>
</property>

<property>
<name>dfs.journalnode.edits.dir</name>
<value>/zn/users/hdpusr01/hadoop/dfs/jn</value>
</property>

<property>
<name>dfs.journalnode.rpc-address</name>
<value>0.0.0.0:8485</value>
</property>

<property>
<name>dfs.journalnode.http-address</name>
<value>0.0.0.0:8480</value>
</property>

<property>
<name>dfs.journalnode.https-address</name>
<value>0.0.0.0:8481</value>
</property>

<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>

<property>
<name>dfs.client.failover.proxy.provider.hdpcls1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>

<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>

<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/zn/users/hdpusr01/.ssh/id_rsa</value>
</property>

<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value>file:/zn/users/hdpusr01/hadoop/dfs/dn/data01</value>
</property>
</configuration>

--修改mapred-site.xml
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop> vi mapred-site.xml
<configuration>

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

--修改yarn-site.xml
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop> vi yarn-site.xml
<configuration>

<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>

<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>rmcls1</value>
</property>

<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>

<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>HBS01</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>HBS02</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>

<property>
<name>yarn.resourcemanager.zk-address</name>
<value>HBS04:29181,HBS05:29181,HBS06:29181</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

--修改slaves
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop> vi slaves
HBS03
HBS04
HBS05
HBS06

--创建hadoop目录
hdpusr01@HBS01:/zn/users/hdpusr01> mkdir -p hadoop/{tmp,dfs/jn,dfs/dn/data01}

--将配置好的hadoop复制到其他节点
hdpusr01@HBS01:/zn/users/hdpusr01> tar cvf hdp.tar hadoop
hdpusr01@HBS04:/zn/users/hdpusr01> scp hdp.tar HBS02:/zn/users/hdpusr01
hdpusr01@HBS04:/zn/users/hdpusr01> scp hdp.tar HBS03:/zn/users/hdpusr01
hdpusr01@HBS04:/zn/users/hdpusr01> scp hdp.tar HBS04:/zn/users/hdpusr01
hdpusr01@HBS04:/zn/users/hdpusr01> scp hdp.tar HBS05:/zn/users/hdpusr01
hdpusr01@HBS04:/zn/users/hdpusr01> scp hdp.tar HBS06:/zn/users/hdpusr01

--解压hadoop包（AI-OPT-HBS02 - HBS05）
tar xvf hdp.tar

--查看zookeeper集群状态（AI-OPT-HBS04 - HBS06）
hdpusr01@HBS04:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /zn/users/hdpusr01/zookeeper/conf/zoo.cfg
Mode: follower

hdpusr01@HBS05:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /zn/users/hdpusr01/zookeeper/conf/zoo.cfg
Mode: leader

hdpusr01@HBS06:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /zn/users/hdpusr01/zookeeper/conf/zoo.cfg
Mode: follower

--启动journalnode（AI-OPT-HBS04 - HBS06）
hdpusr01@HBS04:/zn/users/hdpusr01/hadoop/sbin> ./hadoop-daemon.sh start journalnode
hdpusr01@HBS04:/zn/users/hdpusr01/hadoop/sbin> jps
27035 QuorumPeerMain
28365 JournalNode --正常的情况可以看到该进程

--格式化HDFS（AI-OPT-HBS01）
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hdfs namenode -format
15/12/18 13:34:35 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1518459121-10.46.52.30-1450416875417
15/12/18 13:34:35 INFO common.Storage: Storage directory /zn/users/hdpusr01/hadoop/tmp/dfs/name has been successfully formatted.
15/12/18 13:34:35 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
15/12/18 13:34:35 INFO util.ExitUtil: Exiting with status 0

--拷贝/zn/users/hdpusr01/hadoop/tmp（AI-OPT-HBS01）
hdpusr01@HBS01:/zn/users/hdpusr01> scp -r /zn/users/hdpusr01/hadoop/tmp/* HBS02:/zn/users/hdpusr01/hadoop/tmp

--格式化ZK（AI-OPT-HBS01）
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hdfs zkfc -formatZK
15/12/18 13:49:04 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=HBS04:29181,HBS05:29181,HBS06:29181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@6456c5f4
15/12/18 13:49:04 INFO zookeeper.ClientCnxn: Opening socket connection to server HBS05/10.46.52.34:29181. Will not attempt to authenticate using SASL (unknown error)
15/12/18 13:49:04 INFO zookeeper.ClientCnxn: Socket connection established to HBS05/10.46.52.34:29181, initiating session
15/12/18 13:49:04 INFO zookeeper.ClientCnxn: Session establishment complete on server HBS05/10.46.52.34:29181, sessionid = 0x251b09471e80001, negotiated timeout = 80000
15/12/18 13:49:04 INFO ha.ActiveStandbyElector: Session connected.
15/12/18 13:49:04 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/hdpcls1 in ZK.
15/12/18 13:49:04 INFO zookeeper.ZooKeeper: Session: 0x251b09471e80001 closed
15/12/18 13:49:04 INFO zookeeper.ClientCnxn: EventThread shut down

/* 在格式化ZK后zookeeper中多出的znode */
[zk: 10.46.52.34:29181(CONNECTED) 0] ls /
[zookeeper]
[zk: 10.46.52.34:29181(CONNECTED) 7] ls /
[hadoop-ha, zookeeper]
[zk: 10.46.52.34:29181(CONNECTED) 8] ls /hadoop-ha
[hdpcls1]
[zk: 10.46.52.34:29181(CONNECTED) 9] ls /hadoop-ha/hdpcls1
[]

--启动HDFS（AI-OPT-HBS01）
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> ./start-dfs.sh
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> jps
29964 DFSZKFailoverController
29657 NameNode

hdpusr01@HBS02:/zn/users/hdpusr01/hadoop> jps
28086 NameNode
28185 DFSZKFailoverController

hdpusr01@HBS03:/zn/users/hdpusr01/hadoop/etc/hadoop> jps
27826 DataNode --（AI-OPT-HBS03 - HBS06）

/* 在启动HDFS后zookeeper中多出的znode */
[zk: 10.46.52.34:29181(CONNECTED) 15] ls /hadoop-ha/hdpcls1
[ActiveBreadCrumb, ActiveStandbyElectorLock]
[zk: 10.46.52.34:29181(CONNECTED) 16] ls /hadoop-ha/hdpcls1/ActiveBreadCrumb
[]
[zk: 10.46.52.34:29181(CONNECTED) 17] ls /hadoop-ha/hdpcls1/ActiveStandbyElectorLock
[]
[zk: 10.46.52.34:29181(CONNECTED) 18] get /hadoop-ha/hdpcls1/ActiveBreadCrumb

hdpcls1nn1
HBS01 ?E(?>
cZxid = 0x10000000a
ctime = Fri Dec 18 13:55:41 CST 2015
mZxid = 0x10000000a
mtime = Fri Dec 18 13:55:41 CST 2015
pZxid = 0x10000000a
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 34
numChildren = 0
[zk: 10.46.52.34:29181(CONNECTED) 19] get /hadoop-ha/hdpcls1/ActiveStandbyElectorLock

hdpcls1nn1
HBS01 ?E(?>
cZxid = 0x100000008
ctime = Fri Dec 18 13:55:41 CST 2015
mZxid = 0x100000008
mtime = Fri Dec 18 13:55:41 CST 2015
pZxid = 0x100000008
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x351b09482220000
dataLength = 34
numChildren = 0

--启动YARN(HBS01由于resourcemanager进程占用资源较多，机器充足的情况下可以考虑将其与namenode进程分开）
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> ./start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-resourcemanager-HBS01.out
HBS03: starting nodemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-nodemanager-HBS03.out
HBS04: starting nodemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-nodemanager-HBS04.out
HBS05: starting nodemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-nodemanager-HBS05.out
HBS06: starting nodemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-nodemanager-HBS06.out
注：这个时候主节点上的resourcemanager进程还没启动，从节点上的nodemanager进程已经启动
hdpusr01@HBS03:/zn/users/hdpusr01/hadoop/etc/hadoop> jps
28004 NodeManager --（AI-OPT-HBS03 - HBS06）

--启动resourcemanager
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> ./yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-resourcemanager-HBS01.out
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> jps
30921 ResourceManager --终于看到这个进程了
注：主机AI-OPT-HBS02上的resourcemanager还没启动，需要手工启动
hdpusr01@HBS02:/zn/users/hdpusr01/hadoop/sbin> ./yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-resourcemanager-HBS02.out
hdpusr01@HBS02:/zn/users/hdpusr01/hadoop/sbin> jps
28945 ResourceManager

/* 在启动YARN和resourcemanager后zookeeper中多出的znode */
[zk: 10.46.52.34:29181(CONNECTED) 22] ls /
[rmstore, yarn-leader-election, hadoop-ha, zookeeper]
[zk: 10.46.52.34:29181(CONNECTED) 23] ls /rmstore
[ZKRMStateRoot]
[zk: 10.46.52.34:29181(CONNECTED) 24] ls /rmstore/ZKRMStateRoot
[AMRMTokenSecretManagerRoot, RMAppRoot, RMVersionNode, RMDTSecretManagerRoot]
[zk: 10.46.52.34:29181(CONNECTED) 25] ls /yarn-leader-election
[rmcls1]
[zk: 10.46.52.34:29181(CONNECTED) 26] ls /yarn-leader-election/rmcls1
[ActiveBreadCrumb, ActiveStandbyElectorLock]

--通过命令查看集群状态
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hdfs dfsadmin -report

5.安装HBase
5.1 安装HBase集群（AI-OPT-HBS01 - HBS06）
--选择一台主机操作（AI-OPT-HBS01）
[root@HBS01 hbsusr01]# tar xvf /opt/software/hbase-1.1.2.tgz
[root@HBS01 hbsusr01]# mv hbase-1.1.2 hbase
[root@HBS01 hbsusr01]# chown -R hbsusr01:hbase hbase

--设置集群节点文件
hbsusr01@HBS01:/zn/users/hbsusr01/hbase/conf> vi regionservers
HBS03
HBS04
HBS05
HBS06

--设置hbase配置文件
hbsusr01@HBS01:/zn/users/hbsusr01/hbase/conf> vi hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://HBS01:8920/hbase</value>
</property>

<property>
<name>hbase.master.port</name>
<value>60900</value>
</property>

<property>
<name>hbase.regionserver.port</name>
<value>60920</value>
</property>

<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>

<property>
<name>hbase.zookeeper.quorum</name>
<value>HBS04,HBS05,HBS06</value>
</property>

<property>
<name>hbase.tmp.dir</name>
<value>/zn/users/hbsusr01/hbase/tmp</value>
</property>

<property>
<name>hbase.zookeeper.peerport</name>
<value>29888</value>
</property>

<property>
<name>hbase.zookeeper.leaderport</name>
<value>39888</value>
</property>

<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>29181</value>
</property>

<property>
<name>hbase.rest.port</name>
<value>8980</value>
</property>
</configuration>

--设置hbase环境
hbsusr01@HBS01:/zn/users/hbsusr01/hbase/conf> vi hbase-env.sh
export HBASE_CONF_DIR=/zn/users/hbsusr01/hbase/conf
export HBASE_PID_DIR=/zn/users/hbsusr01/hbase/pid
export HBASE_LOG_DIR=/zn/users/hbsusr01/hbase/logs

# export HBASE_MANAGES_ZK=true
export HBASE_MANAGES_ZK=false

--创建相关目录
hbsusr01@HBS01:/zn/users/hbsusr01> mkdir -p hbase/{conf,pid,logs,tmp}

--拷贝文件到其他节点并解压
hbsusr01@HBS01:/zn/users/hbsusr01> tar cvf hbase.tar hbase
hbsusr01@HBS01:/zn/users/hbsusr01> scp hbase.tar HBS03:/zn/users/hbsusr01
hbsusr01@HBS01:/zn/users/hbsusr01> scp hbase.tar HBS04:/zn/users/hbsusr01
hbsusr01@HBS01:/zn/users/hbsusr01> scp hbase.tar HBS05:/zn/users/hbsusr01
hbsusr01@HBS01:/zn/users/hbsusr01> scp hbase.tar HBS06:/zn/users/hbsusr01

--配置ssh互信
su - hbsusr01
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
chmod 600 .ssh/authorized_keys

--选择一台master操作(HBS01)
ssh HBS02 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS03 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS04 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS05 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS06 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

scp ~/.ssh/authorized_keys HBS02:/zn/users/hbsusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS03:/zn/users/hbsusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS04:/zn/users/hbsusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS05:/zn/users/hbsusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS06:/zn/users/hbsusr01/.ssh/authorized_keys

--验证配置是否生效
ssh HBS01 date
ssh HBS02 date
ssh HBS03 date
ssh HBS04 date
ssh HBS05 date
ssh HBS06 date

--解压hadoop包（AI-OPT-HBS03 - HBS06）
tar xvf hbase.tar

--启动hbase
start-hbase.sh
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hbsusr01, access=WRITE, inode="/":hdpusr01:supergroup:drwxr-xr-x

解决方法：hadoop fs添加目录和权限
hdpusr01@HBS01:/zn/users/hdpusr01> hadoop fs -mkdir /hbase
-mkdir: java.net.UnknownHostException: host-10-1-241-18

临时解决：
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hadoop fs -mkdir /hbase
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hadoop fs -chown hbsusr01:hbase /hbase

/*
解决方法：配置环境变量
su - hdpusr01
export HADOOP_CONF_DIR=$HOME/hadoop/etc/hadoop
#export HADOOP_PID_DIR=$HOME/hadoop/pid
export HADOOP_LOG_DIR=$HOME/hadoop/logs
export HADOOP_LIBEXEC_DIR=$HOME/hadoop/libexec
export HADOOP_PREFIX=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export HADOOP_LIB=${HADOOP_HOME}/lib
export HADOOP_LIBRARY_PATH=${HADOOP_HOME}/lib/native
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_YARN_USER=pochdp04
export YARN_CONF_DIR=${HADOOP_CONF_DIR}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_YARN_HOME=${HADOOP_HOME}
export YARN_LOG_DIR=${HADOOP_LOG_DIR}
export YARN_PID_DIR=${HADOOP_CONF_DIR}/../yarn
*/

su - hbsusr01
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/zn/users/hdpusr01/hadoop/lib/native
export HBASE_HOME=/zn/users/hbsusr01/hbase
export HBASE_CONF_DIR=$HBASE_HOME/conf
export HBASE_LOG_DIR=$HBASE_HOME/logs
export HBASE_PID_DIR=$HBASE_HOME/pid
export PATH=$PATH:$HBASE_HOME/bin

http://blog.itpub.net/20777547/viewspace-1745820/
http://www.aboutyun.com/thread-11909-1-1.html
http://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop-yarn/

***********************************************hadoop伪分布式部署**********************************************
1.创建用户以及配置主机名
mkdir -p /zn/users
useradd -u 451 -g hadoop -G ibss -d /zn/users/hdpusr02 -m hdpusr02
passwd hdpusr02 --stdin <<< Linuhdp_0805
echo -e "\n. ~puwadm/wprofile\n" >> ~hdpusr02/.bash_profile

2.配置ssh互相
su - hdpusr02
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh AI-UAT-OPT-HBASE03 date

3.解压文件
tar xvf /opt/software/hadoop-2.6.0.tgz
mv hadoop-2.6.0/ hadoop

4.配置hadoop
vi hadoop-env.sh
export JAVA_HOME=/opt/freeware/jdk1.7.0_79

vi core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/zn/users/hdpusr01/hadoop/tmp</value>
</property>

<property>
<name>fs.defaultFS</name>
<value>hdfs://AI-UAT-OPT-HBASE03:9000</value>
</property>
</configuration>

vi mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

vi yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

vi hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.namenode.name.dir</name>
<value>file:/zn/users/hdpusr02/hadoop/dfs/name</value>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value>file:/zn/users/hdpusr02/hadoop/dfs/data</value>
</property>
</configuration>

vi slaves
AI-UAT-OPT-HBASE03

5.创建相应目录
mkdir -p hadoop/dfs/{name,data}

6.启动hadoop
bin/hdfs namenode -format
sbin/start-dfs.sh
sbin/start-yarn.sh