hadoop+hbase

[code="java"]
131做master ，91,133做salve
131操作

192.168.0.131 chupiao
192.168.0.91 chupiao2
192.168.0.133 rycdb

useradd hadoop

一.ssh免密码登陆
ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
b0:b0:84:bb:ad:27:15:c4:0f:56:4c:1e:68:cb:73:53 hadoop@chupiao

[hadoop@chupiao ~]$ cd .ssh/
[hadoop@chupiao ~]$ll
[hadoop@chupiao .ssh]$ cp id_rsa.pub authorized_keys
[hadoop@chupiao .ssh]$ ssh chupiao
The authenticity of host 'chupiao (192.168.0.131)' can't be established.
RSA key fingerprint is 3a:a1:ae:c8:1c:a0:15:ef:9e:3d:87:dd:89:9f:f1:93.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'chupiao,192.168.0.131' (RSA) to the list of known hosts.
Last login: Thu Jan 10 10:03:41 2013 from localhost.localdomain

输入 yes 来继续。这会把该服务器添加到你的已知主机的列表中
发现链接成功，并且无需密码。

在 91，133机器上分别操作
# useradd hadoop
# su - hadoop
[hadoop@chupiao2 ~]$ ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
06:88:7e:7c:c8:64:25:4c:09:b9:04:b2:03:86:ab:df hadoop@chupiao2
cd .ssh
[hadoop@chupiao .ssh]$ chmod 644 authorized_keys
在131机器上执行，将authorized_keys传输到91,133

[hadoop@chupiao .ssh]$ scp authorized_keys 192.168.0.133:/home/hadoop/.ssh
测试：ssh 91或者ssh 133（第一次需要输入yes）。
如果不须要输入密码则配置成功，如果还须要请检查上面的配置能不能正确。

在三台机器上，下载hadoop并解压

vi hadoop-env.sh
export JAVA_HOME=/usr/local/jdk1.6.0_32
export HADOOP_HOME=/home/hadoop/hadoop-2.0.2-alpha
vi /home/hadoop/hadoop-2.0.2-alpha/etc/hadoop/core-site.xml


<configuration>
<property>
<!-- 用于dfs命令模块中指定默认的文件系统协议 -->
<name>fs.default.name</name>
<value>hdfs://192.168.0.131:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/tmp/var</value>
<!-- hadoop错误INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1 -->

</property>
</configuration>

1）fs.default.name是NameNode的URI。hdfs://主机名:端口/
2）hadoop.tmp.dir ：Hadoop的默认临时路径，这个最好配置，
如果在新增节点或者其他情况下莫名其妙的DataNode启动不了，
就删除此文件中的tmp目录即可。不过如果删除了NameNode机器的此目录，
那么就需要重新执行NameNode格式化的命令。

conf/mapred-site.xml

<?xmlversion="1.0"?>
<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>192.168.0.131:9001</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/hadoop/tmp/var</value>
</property>
</configuration>

1）mapred.job.tracker是JobTracker的主机（或者IP）和端口。主机:端口。
conf/hdfs-site.xml

<?xmlversion="1.0"?>
<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/name1, /home/hadoop/name2</value> #hadoop的name目录路径
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/data1, /home/hadoop/data2</value>
</property>
<property>
<name>dfs.replication</name>
<!-- 我们的集群又两个结点，所以rep两份 -->
<value>2</value>
</property>
</configuration>

1） dfs.name.dir是NameNode持久存储名字空间及事务日志的本地文件系统路径。
当这个值是一个逗号分割的目录列表时，nametable数据将会被复制到所有目录中做冗余备份。
2） dfs.data.dir是DataNode存放块数据的本地文件系统路径，逗号分割的列表。 当这个值是逗号分割的目录列表时，数据将被存储在所有目录下，通常分布在不同设备上。
3）dfs.replication是数据需要备份的数量，默认是3，如果此数大于集群的机器数会出错。

注意：此处的name1、name2、data1、data2目录不能预先创建，hadoop格式化时会自动创建，如果预先创建反而会有问题。

vi Masters

192.168.0.131
vi
Slaves

192.168.0.133
192.168.0.91


启动

sbin/
start-all.sh

查看是否启动

/bin/
./hadoop dfs -ls
　　如果没有错误提示或者出现文件列表，那么恭喜你，Hadoop成功启动了

http://www.cnblogs.com/aniuer/archive/2012/07/16/2594448.html
最新版本的安装文档

[hadoop@hadoop ~]$ vi /home/hadoop/HadoopInstall/hadoop-conf/hadoop-env.sh
指定JAVA_HOME和HADOOP_HOME
export JAVA_HOME=/usr/local/jdk1.6.0_32
export HADOOP_HOME=/home/hadoop/hadoop-2.0.2-alpha

export PATH=$PATH:/home/hadoop/hadoop-0.22.0/bin

由于Hadoop用ssh 通信，因此先进行免密码登录设定，
$ apt-get install ssh
$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub>> ~/.ssh/authorized_keys
chmod 600 authorized_keys
或者 cp id_rsa.pub authorized_keys

$ ssh localhost
完成后请登入确认不用输入密码，（第一次登入需按enter键，第二次就可以直接登入到系统。
~$ ssh localhost
~$ exit
~$ ssh localhost
~$ exit
如果mapred启动不起来 用 start-mapred.sh试一下，请查看start-all 里面的各环境变量

HADOOP_MAPRED_HOME

http://blog.csdn.net/hguisu/article/details/7237395

http://www.cnblogs.com/maczpc/archive/2010/01/17/1650191.htm

./hadoop dfs -mkdir /input
./hadoop fs -ls /
./hadoop dfs -put ~/test-txt /input

./hadoop dfs -ls /input/test-txt/file2.txt
./hadoop fs -cat /input/test-txt/file2.txt
./hadoop dfs -cat /input/test-txt/file2.txt
hadoop 命令
http://blog.csdn.net/cheungmine/article/details/7744487

zookeeper 安装
cp zoo_sample.cfg zoo.cfg
http://www.linuxidc.com/Linux/2012-07/65607.htm

vi zoo.cfg
dataDir=/home/hadoop/zookeeper-3.4.3/data
server.1=192.168.0.131:2888:3888
server.2=192.168.0.91:2888:3888
server.3=192.168.0.133:2888:3888
#dataLogDir=/home/hadoop/zookeeper/log # 日志保存路径

/home/hadoop/zookeeper-3.4.3/data
vi myid
1

在各节点上分别启动
./zkServer.sh start
各节点测试测试
./zkCli.sh 192.168.0.91:2181

http://www.linuxidc.com/Linux/2012-07/65607.htm

export ZOOKEEPER_INSTALL=/home/hadoop/zookeeper-3.4.3
export PATH=$PATH:$ZOOKEEPER_INSTALL/bin

hbase 安装
http://blog.csdn.net/shi1122/article/details/7676488


1、修改conf/hbase-env.sh

export JAVA_HOME=/usr/local/jdk1.6.0_32
export HBASE_CLASSPATH=/home/hadoop/hbase-0.94.0/conf
export HBASE_MANAGES_ZK=false

2、修改hbase-site.xml，增加以下内容

<property>

<name>hbase.rootdir</name>

<value>hdfs://192.168.0.131:9000/hbase</value>(注：这里须hadoop-config/core-site.xml中的fs.default.name保持一致）

</property>

<property>

<name>hbase.cluster.distributed</name>

<value>true</value>

</property>

<property>

<name>hbase.zookeeper.property.clientPort</name>

<value>2181</value>(注：这里的端口需要和zookeeper的端口配置值一样）

<description>Property from ZooKeeper's config zoo.cfg.</description>
</property>

vi regionservers
192.168.0.131
192.168.0.91
192.168.0.133

hbase 启动出现版本不一样的错误：
org.apache.Hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown
将HBase解压缩包中lib的Hadoop Core jar文件替换为当前你所使用的Hadoop版本即可
2.2 版本是
/home/hadoop/hadoop-2.0.2-alpha/share/hadoop/common
hadoop-comm
2013-01-11 15:55:41,202 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster

3、把/home/frank/HadoopInstall/hadoop-config/hdfs-site.xml文件拷贝至hbase的conf文件夹下

4、把${ZOOKEEPER_HOME}/conf/zoo.cfg拷贝至hbase的conf文件夹下

5、在conf/regionservers中添加hadoop-config/conf/slaves中所有的datanode节点。

6、删除/hbase-0.90.2/lib/hadoop-core-0.20-append-r1056497.jar

拷贝/hadoop-0.20.2/hadoop-0.20.0-core.jar到/hbase-0.90.2/lib/


java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1740)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:146)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1754)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.security.authentication.util.KerberosName
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:212)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:275)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
at org.apache.hadoop.hbase.security.User.call(User.java:586)
at org.apache.hadoop.hbase.security.User.callStatic(User.java:576)
at org.apache.hadoop.hbase.security.User.access$400(User.java:50)
at org.apache.hadoop.hbase.security.User$SecureHadoopUser.isSecurityEnabled(User.java:564)
at org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:535)
at org.apache.hadoop.hbase.security.User.login(User.java:198)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:267)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1735)
... 5 more


Server IPC version 5 cannot communicate with client version 4

版本不同引起的，可以将 hadoop 里面的相关jar包放在hbase的lib里面
cd bin
hbase shell
hbase(main):001:0> help
hbase(main):002:0> status
hbase(main):003:0> version

java 程序如何操作hbase
http://blog.csdn.net/shi1122/article/details/7676488

http://blog.csdn.net/karen_wang/article/details/6283729

[/code]