hadoop集群配置

[size=large][b]1、准备工作[/b][/size]
三台机器
report1.ka.hadoop.corp.koubei.net(master)
report2.ka.hadoop.corp.koubei.net(slave)
report3.ka.hadoop.corp.koubei.net(slave)
http://www.apache.org/dist//hadoop/core/hadoop-0.18.3/
http://www.apache.org/dist//hadoop/hive/hive-0.5.0/
[size=large][b]2、SSH配置[/b][/size]
Hadoop需要namenode到datanode的无密码SSH登录
# cd /root/.ssh/
# ssh-keygen -t rsa
# touch authorized_keys
# cat id_rsa.pub >>authorized_keys
将report1 ./.ssh/id_rsa.pub内容追加 report2、report3 authorized_keys文件末尾
[b][size=large]3、jdk安装配置[/size][/b]
由于redhat中自带安装了jdk1.4的，所以在安装jdk1.6前先把jdk1.4卸载
#rpm -qa | grep gcj
#rpm -e --nodeps java-1.4.2-gcj-compat-1.4.2.0-40jpp.115
# ./jdk-linux.bin
[b].bash_profile[/b]
export JAVA_HOME=/opt/jdk1.6.0_21
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
# source .bash_profile
[size=large][b]4、hadoop安装配置[/b][/size]
#cd /opt
#tar -xzvf hadoop-0.18.3.tar.gz
[b].bash_profile[/b]
export HADOOP_INSTALL=/opt/hadoop-0.18.3
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_INSTALL/bin
hadoop-env.sh
export JAVA_HOME=/opt/jdk1.6.0_21
[b]hadoop-site.xml[/b]
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://report1.ka.hadoop.corp.koubei.net</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>report1.ka.hadoop.corp.koubei.net:8021</value>
</property>
</configuration>
report1.ka.hadoop.corp.koubei.net配置namenode及datanode
[b]masters[/b]
report1.ka.hadoop.corp.koubei.net
[b]slaves [/b]
report2.ka.hadoop.corp.koubei.net
report3.ka.hadoop.corp.koubei.net
# hadoop namenode -format
#hadoop fs -ls /tmp
# hadoop fs -put test.txt hdfs://report1.ka.hadoop.corp.koubei.net/tmp/test.txt
[size=large][b]5、hive安装配置[/b][/size]
# tar -zxvf hive-0.5.0-dev.tar.gz
[b].bash_profile[/b]
export HIVE_HOME=/opt/hive-0.5.0-dev
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_INSTALL/bin:$HIVE_HOME/bin
#source .bash_profile