Hadoop,Hbase,Zookeeper在虚拟机单节点中的整合

最新推荐文章于 2020-08-10 08:41:49 发布

haohouhou

最新推荐文章于 2020-08-10 08:41:49 发布

阅读量205

点赞数

分类专栏： hadoop 文章标签： hadoop hbase zookeeper 虚拟机 centos

hadoop 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

近日整合Hadoop,Hbase,Zookeeper时遇到不少问题，经过一番苦战后终于整合成功，现将整合中遇到的一些问题整理一下。

[b]前置说明：[/b]
版本：
JDK: 1.6.*
Hadoop: 0.20.2
Hbase: 0.90.4
Zookkeeper: 3.4.3
centOS: CentOS release 5.3

节点数： 1 （虚拟机）

[b]安装配置：[/b]
一、安装JDK

二、配置host
修改配置 /etc/hosts 如下：
192.168.0.23 hadoopName
此hostname为hadoop、Hbase、Zookeeper配置文件中共同使用。这一步非常重要。也会涉及到windows中eclipse runonhadoop插件的运行。

二、安装Hadoop
Hadoop安装比较容易，请参考《hadoop权威指南》、陆嘉恒版《hadoop实战》。

三、安装Zookeeper
运行模式为：[b]集群伪分布模式[/b]
1、解压Zookeeper，将$ZK_INSTALL/bin加入环境变量。
2、拷贝$ZK_INSTALL/conf下的zoo.example.cfg，分别复制为zoo1.cfg、zoo2.cfg、zoo3.cfg
3、配置zoo*.cfg：
zoo1.cfg


# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/usr/local/zookeeper-3.4.3/snapshot/d_1
# the port at which the clients will connect
clientPort=2181
#the location of the log file
dataLogDir=/usr/local/zookeeper-3.4.3/logs
server.[color=red]1[/color]=hadoopName:2887:3887
server.[color=red]2[/color]=hadoopName:2888:3888
server.[color=red]3[/color]=hadoopName:2889:3889

zoo2.cfg


# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/usr/local/zookeeper-3.4.3/snapshot/d_2
# the port at which the clients will connect
clientPort=2182
#the location of the log file
dataLogDir=/usr/local/zookeeper-3.4.3/logs
server.[color=red]1[/color]=hadoopName:2887:3887
server.[color=red]2[/color]=hadoopName:2888:3888
server.[color=red]3[/color]=hadoopName:2889:3889

zoo3.cfg


# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/usr/local/zookeeper-3.4.3/snapshot/d_3
# the port at which the clients will connect
clientPort=2183
#the location of the log file
dataLogDir=/usr/local/zookeeper-3.4.3/logs
server.[color=red]1[/color]=hadoopName:2887:3887
server.[color=red]2[/color]=hadoopName:2888:3888
server.[color=red]3[/color]=hadoopName:2889:3889

4、在zoo*.cfg中的dataDir指定的目录下，[color=red]新建myid文件[/color]。
例如：$ZK_INSTALL/snapshot/d_1下，新建myid。在myid文件中输入1。表示为server.1。
如果为snapshot/d_2，则myid文件中的内容为 2，依此类推。

5、如果在zoo*.cfg中配置了

 
       #the location of the log file
       dataLogDir=/usr/local/zookeeper-3.4.3/[color=red]logs[/color]

如果$ZK_INSTALL下不存在logs目录，也会报错。所以需要手动创建logs文件夹。
报错异常信息如下：


2012-02-26 00:06:47,364 [myid:] - INFO  [main:QuorumPeerConfig@101] - Reading configuration from: ../conf/zoo3.cfg
2012-02-26 00:06:47,372 [myid:] - ERROR [main:QuorumPeerMain@85] - Invalid config, exiting abnormally
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing ../conf/zoo3.cfg
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:121)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: [color=red]java.lang.IllegalArgumentException: dataLogDir /usr/local/zookeeper-3.4.3/logs is missing.[/color]
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:247)
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:117)
        ... 2 more
Invalid config, exiting abnormally

6、现在就可以启动zookeeper了：
zkServer.sh start zoo1.cfg
zkServer.sh start zoo2.cfg
zkServer.sh start zoo3.cfg
然后输入jps，查看进程：
# jps
4208 SecondaryNameNode
4587 QuorumPeerMain
4498 QuorumPeerMain
4273 JobTracker
4014 NameNode
4106 DataNode
4617 Jps
4547 QuorumPeerMain
4424 TaskTracker

有3个QuorumPeerMain进程启动了，zookeeper启动成功。还可以使用zookeeper自带的四字命令进行启动成功的测试：
# echo ruok | nc hadoopName 2181
imok
返回了imok，证明启动成功。如果什么都没有返回，证明启动失败。

四、安装Hbase
1、解压Hbase后，将$HBASE_INSTALL/bin加入环境变量。
2、配置$HBASE_INSTALL/conf/hbase-env.sh：
export JAVA_HOME=$YOUR_JAVA_HOME
# Tell HBase whether it should manage it's own instance of Zookeeper or not.
export HBASE_MANAGES_ZK=false <==默认为true，我们不使用hbase自带的zookeeper实例。

3、配置hbase-site.xml


<configuration>
<property>
        <name>hbase.rootdir</name>
        <value>hdfs://[color=red]hadoopName[/color]:9000/hbase</value>
</property>
<property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
</property>
<property>
        <name>dfs.replication</name>
        <value>1</value>
</property>
<property>
        <name>hbase.zookeeper.quorum</name>
        <value>[color=red]hadoopName[/color]</value>
</property>
<property>
        <name>zookeeper.session.timeout</name>
        <value>60000</value>
</property>

5、将$HBASE_INSTALL/lib下的hadoop-core-0.20-append-r1056497.jar删除，替换为$HADOOP_INSTALL/hadoop-0.20.2-core.jar
如果不替换，则在调用start-hbase.sh时HMaster无法启动。
报错异常如下：


      12/02/26 00:35:02 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoopname/192.168.0.23:2181, sessionid = 0x135b544b002000f, negotiated timeout = 40000
12/02/26 00:35:02 INFO client.HConnectionManager$HConnectionImplementation: ZooKeeper available but no active master location found
12/02/26 00:35:02 INFO client.HConnectionManager$HConnectionImplementation: getMaster attempt 0 of 1 failed; no more retrying.
org.apache.hadoop.hbase.MasterNotRunningException
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:357)
        at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:94)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.stopMaster(HMasterCommandLine.java:163)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1078)
12/02/26 00:35:02 ERROR master.HMasterCommandLine: Master not running

ps：在hbase官方说明中有另外一种方法，小生暂时未试验过。[url]http://hbase.apache.org/book.html#hadoop[/url]

6、这时我们再启动start-hbase.sh，就会成功啦！
# start-hbase.sh
starting master, logging to /usr/local/hbase-0.90.4/logs/hbase-root-master-hadoopname.out
hadoopName: starting regionserver, logging to /usr/local/hbase-0.90.4/logs/hbase-root-regionserver-hadoopname.out
# jps
5004 HMaster
5101 HRegionServer
4208 SecondaryNameNode
5181 Jps
4587 QuorumPeerMain
4498 QuorumPeerMain
4273 JobTracker
4014 NameNode
4106 DataNode
4547 QuorumPeerMain
4424 TaskTracker

至此，hadoop、hbase、zookeeper在虚拟机下的整合就完毕了。

[b]总结：[/b]
从整合的过程中可以感受到，hadoop的问题较少，而hbase和zk问题较多。比如zk的zoo.cfg配置参数中，dataDir指定的目录如果未创建，则zk会自动创建；但是dataLogDir参数指定的路径却不会自动创建。
最后，如果本文中有什么遗漏或不正确的地方，还请各位朋友斧正。在下感激不尽！

haohouhou

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
Hadoop,Hbase,Zookeeper在虚拟机单节点中的整合

近日整合Hadoop,Hbase,Zookeeper时遇到不少问题，经过一番苦战后终于整合成功，现将整合中遇到的一些问题整理一下。[b]前置说明：[/b]版本：JDK: 1.6.*Hadoop: 0.20.2Hbase: 0.90.4Zookkeeper: 3.4.3centOS: CentOS release 5.3 节点数： 1 （虚拟机）...
复制链接

扫一扫