系统:ubuntu10.10
用户:xcloud
安装路径:/opt
apache下载地址:http://mirror.bjtu.edu.cn/apache//
一.JDK
1.解压安装
xcloud@xcloud:/opt$ sudo chmod +x jdk-6u30-linux-i586.binxcloud@xcloud:/opt$ sudo -s ./jdk-6u30-linux-i586.bin
xcloud@xcloud:/opt$ ln -s jdk1.6.0_30 jdk6
xcloud@xcloud:/opt$ sudo chown xcloud -R jdk*
xcloud@xcloud:/opt$ sudo chmod 775 -R jdk*
2.设置环境变量
xcloud@xcloud:/opt$ sudo gedit /etc/environmentJAVA_HOME=/opt/jdk6
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/jdk6/bin
xcloud@xcloud:/opt$ source /etc/environment
3.测试
xcloud@xcloud:/opt$ java -versionjava version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) Server VM (build 20.5-b03, mixed mode)
二.Eclipse
1.安装
xcloud@xcloud:/opt$ sudo tar zxvf eclipse-jee-indigo-SR1-linux-gtk.tar.gzxcloud@xcloud:/opt$ sudo chown xcloud -R eclipse
xcloud@xcloud:/opt$ sudo chmod 775 -R eclipse
xcloud@xcloud:/opt$ cd eclipse/
xcloud@xcloud:/opt/eclipse$ ./eclipse &
三.Tomcat
1.安装
xcloud@xcloud:/opt$ sudo unzip apache-tomcat-6.0.35.zipxcloud@xcloud:/opt$ sudo mv apache-tomcat-6.0.35 tomcat6
xcloud@xcloud:/opt$ sudo chown xcloud -R tomcat6
xcloud@xcloud:/opt$ sudo chmod 775 -R tomcat6
xcloud@xcloud:/opt$ sudo gedit /etc/environment
CATALINA_HOME=/opt/tomcat6
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/jdk6/bin:/opt/tomcat6/bin
xcloud@xcloud:/opt$ source /etc/environment
四.Hadoop
1.解压
xcloud@xcloud:/opt/cloudera$ sudo tar zxvf hadoop-0.20.2-cdh3u2.tar.gzxcloud@xcloud:/opt/cloudera$ sudo mv hadoop-0.20.2-cdh3u2 ..
xcloud@xcloud:/opt$ sudo mv hadoop-0.20.2-cdh3u2 hadoop
2.修改配置文件
xcloud@xcloud:/opt$ cd hadoop/conf
xcloud@xcloud:/opt/hadoop/conf$ sudo gedit hadoop-env.sh
+
export JAVA_HOME=/opt/jdk6
xcloud@xcloud:/opt$ sudo chown xcloud -R hadoop
xcloud@xcloud:/opt$ sudo chmod 775 -R hadoop
3.安装SSH/修改无密码验证SSH登录模式
sudo apt-get install openssh-serversudo apt-get install openssh-client
xcloud@xcloud:/opt/hadoop/dataRoot/output$ sudo gedit /etc/hosts
xcloud@xcloud:~$ sudo apt-get install openssh-server
xcloud@xcloud:~$ sudo apt-get install openssh-client
xcloud@xcloud:~$ ssh-keygen -t rsa
xcloud@xcloud:~$ cd /home/xcloud/.ssh/
xcloud@xcloud:~/.ssh$ cp id_rsa.pub authorized_keys
4.hadoop 配置
主机的配置
gedit /etc/hosts127.0.0.1 localhost
127.0.1.1 xcloud
10.45.46.106 xcloud
xcloud
conf/hadoop-env.sh :
export JAVA_HOME=/opt/jdk6conf/core-site.xml:
<property><name>hadoop.tmp.dir</name>
<value>/opt/hadoop/dataRoot</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://xcloud:9000/</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>
/conf/hdfs-site.xml
<property><name>dfs.name.dir</name>
<value>/opt/hadoop/dataRoot/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/opt/hadoop/dataRoot/data</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
conf/mapred-site.xml:
<property><name>mapred.job.tracker</name>
<value>xcloud:9001</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
conf/masters:
xcloudconf/slaves:
xcloud
/etc/environment
HADOOP_HOME=/opt/hadoop
HBASE_HOME=/opt/hbase
ZOOKEEPER_HOME=/opt/zookeeper
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/jdk6/bin:/opt/tomcat6/bin:/opt/hadoop/bin:/opt/hbase/bin:/opt/zookeeper/bin
HBASE_HOME=/opt/hbase
ZOOKEEPER_HOME=/opt/zookeeper
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/jdk6/bin:/opt/tomcat6/bin:/opt/hadoop/bin:/opt/hbase/bin:/opt/zookeeper/bin
5.启动测试
xcloud@xcloud:/opt/hadoop/bin$start-all.shxcloud@xcloud:/opt/hadoop/bin$netstat -an|grep 9000
xcloud@xcloud:/opt/hadoop/bin$netstat -an|grep 9001
xcloud@xcloud:/opt/hadoop/bin$jps
xcloud@xcloud:/opt/hadoop/bin$hadoop dfs -ls
6.eclipse环境
xcloud@xcloud:/opt/hadoop20/contrib/eclipse-plugin$ cp * /opt/eclipse/plugins/xcloud@xcloud:/opt/hadoop20/contrib/eclipse-plugin$ cd /opt/eclipse/plugins/
xcloud@xcloud:/opt/eclipse/plugins$ mv hadoop-0.20.2-eclipse-plugin.jar hadoop.jar
启动eclipse,进行设置
打开eclipse,看到Project Explorer有个DFS Locations目录
Window > Preferences > Hadoop Map/Reduce 设置安装目录:/opt/hadoop
Window > Show View > Other > MapReduce Tools选择Map/Reduce Locations
在打开到控制台中,点击右边蓝色小象图标按钮> New Hadoop location...
Location name :hadoop
Map/Reduce Master下
Host :xcloud
Port :9001
DFS Master
Port :9000
User name : xcloud
7.问题列表
问题1:
Cannot connect to the Map/Reduce location: hadoopCall to xcloud/127.0.1.1:9001 failed on local exception: java.io.EOFException
解决:
下载 hadoop-0.20.3-dev-eclipse-plugin.jar覆盖原先的 hadoop-0.20.2.jar
http://code.google.com/p/hadoop-eclipse-plugin/downloads/detail?name=hadoop-0.20.3-dev-eclipse-plugin.jar&can=2&q=
参考:http://heipark.iteye.com/blog/786302
问题2:无法启动namenode
通过日志文件进行分析:tail -f /opt/hadoop/logs/hadoop-xcloud-namenode-xcloud.log有用信息:2011-12-22 09:23:45,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 9000, call addBlock(/opt/hadoop20/dataRoot/tmp/mapred/system/jobtracker.info, DFSClient_2118960177) from 127.0.0.1:54034: error: java.io.IOException: File /opt/hadoop20/dataRoot/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /opt/hadoop20/dataRoot/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
解决:
认真看遍 配置文件
删除原来目录
格式化
问题3:
xcloud@xcloud:/opt/hadoop20/bin$ hadoop dfs -lsBad connection to FS. command aborted. exception: Call to xcloud/127.0.0.1:9000 failed on local exception: java.io.EOFException
LOG信息:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /opt/hadoop20/dataRoot/tmp/mapred/system. Name node is in safe mode.
The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe mode will be turned off automatically in 8 seconds.
1、检查防火墙是否关闭?
2、把safemode置于off状态:hadoop dfsadmin -safemode leave
3、也有可能是刚启动,datanode与namenode正在进行状态检测,所以可以等待几分钟。
问题0:
2011-12-22 12:41:33,714 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.java.io.IOException: Missing directory /opt/hadoop/dataRoot/name
解决:
xcloud@xcloud:/opt/hadoop/dataRoot$ mkdir name
问题0——1:
2011-12-22 12:44:41,282 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException: NameNode is not formatted.
解决:
xcloud@xcloud:/opt/hadoop/dataRoot$ hadoop namenode -format
问题:eclipse plugin
Cannot connect to the Map/Reduce location: hadoop_0
Call to xcloud/127.0.0.1:9001 failed on local exception: java.io.EOFException