一、安装Hadoop
<2.1>运行sudo apt-get install ssh/rsync,运行sudo apt-get install openjdk-6-jdk (for jps command)
<2.2>配置ssh本机免口令登录主要有两步:
运行ssh-keygen -t dsa -P '' -f /.ssh/id_dsa
-t用来指定加密算法,可以选择dsa和rsa两种加密方式;
-P用来指定密码,两个单引号表示空密码'';
-f用来指定存放密钥的文件
运行cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
这一步将公钥添加进本机的authorized_keys中,完成这两步后可以ssh localhost验证是否成功。接下来需要进入已经解好的hadoop-1.0.3中进行配置。hadoop的伪分布模式主要需要配置以下几个配置文件:
<3>conf/hbase-env.sh:主要用来配置hadoop的运行环境,这里需要修改JAVA_HOME到你的jdk1.6目录(见黑体)
<4>conf/core-site.xml
这里主要配置fs.default.name(用来指定namenode)和hadoop.tmp.dir(默认的hdfs的tmp目录位置),这里可以不设置hadoop.tmp.dir,那么就会保存在默认的/tmp下,每次重启机器都会丢失数据。
<5>hdfs-site.xml
这里的dfs.replication用来设置每份数据块的副本数目,默认是3,因为我们是在单机上配置的伪分布模式,因此设为1。dfs.name.dir和dfs.data.dir非常重要,用来设置存放hdfs中namenode和datanode数据的本地存放位置。这里如果设置不好,后续会出现多个错误。当然你也可以不设置采用默认的/tmp下的目录,但是同样重启会丢失数据。
<6>mapred-site.xml
然后就是运行测试了,把hadoop-1.0.3/bin加入到/etc/profile中的PATH路径中,方便我们执行Hadoop命令。运行start-all.sh后出现了问题,jps查看namenode无法启动,使用hadoop namenode -format也不能成功,查看日志:
提示我们存储的HDFS目录要么不存在要么没有权限:
FSNamesystem initialization failed.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:Directory /home/hadoop/hdfs/name is in an inconsistent state: storage directory does not exist or is not accessible.
查看发现已经生成了hdfs目录,那么问题就是权限了,将/home下的hadoop目录权限由755设为775,然后重新运行hadoop,成功:
在进行安装hbase之前,我们先来按照官方的方法测试一下伪分布式的hadoop,看看安装是否成功:首先将conf下的所有文件拷贝到hdfs上的input目录中,然后运行jar文件将结果存储到hdfs中的output目录中,最后从output目录中查看结果:
<1> 安装JDK1.6
chmod该文件以775的可执行权限,然后./filename.bin即可;
<2> 安装ssh,并且设置ssh无密码登录hadoop<2.1>运行sudo apt-get install ssh/rsync,运行sudo apt-get install openjdk-6-jdk (for jps command)
<2.2>配置ssh本机免口令登录主要有两步:
运行ssh-keygen -t dsa -P '' -f /.ssh/id_dsa
-t用来指定加密算法,可以选择dsa和rsa两种加密方式;
-P用来指定密码,两个单引号表示空密码'';
-f用来指定存放密钥的文件
运行cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
这一步将公钥添加进本机的authorized_keys中,完成这两步后可以ssh localhost验证是否成功。接下来需要进入已经解好的hadoop-1.0.3中进行配置。hadoop的伪分布模式主要需要配置以下几个配置文件:
<3>conf/hbase-env.sh:主要用来配置hadoop的运行环境,这里需要修改JAVA_HOME到你的jdk1.6目录(见黑体)
点击(此处)折叠或打开
- #Set Hadoop-specific environment variables here.
-
- # The only required environment variable is JAVA_HOME. All others are
- # optional. When running a distributed configuration it is best to
- # set JAVA_HOME in this file, so that it is correctly defined on
- # remote nodes.
-
- # The java implementation to use. Required.
- export JAVA_HOME=/home/hadoop/platform/jdk1.6.0_45
-
- # Extra Java CLASSPATH elements. Optional.
- # export HADOOP_CLASSPATH=
-
- # The maximum amount of heap to use, in MB. Default is 1000.
- # export HADOOP_HEAPSIZE=2000
-
- # Extra Java runtime options. Empty by default.
- # export HADOOP_OPTS=-server
-
- # Command specific options appended to HADOOP_OPTS when specified
- export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
- export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
- export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
- export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
- export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
- # export HADOOP_TASKTRACKER_OPTS=
- # The following applies to multiple commands (fs, dfs, fsck, distcp etc)
- # export HADOOP_CLIENT_OPTS
-
- # Extra ssh options. Empty by default.
- # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"
-
- # Where log files are stored. $HADOOP_HOME/logs by default.
- # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
-
- # File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
- # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
-
- # host:path where hadoop code should be rsync'd from. Unset by default.
- # export HADOOP_MASTER=master:/home/$USER/src/hadoop
-
- # Seconds to sleep between slave commands. Unset by default. This
- # can be useful in large clusters, where, e.g., slave rsyncs can
- # otherwise arrive faster than the master can service them.
- # export HADOOP_SLAVE_SLEEP=0.1
-
- # The directory where pid files are stored. /tmp by default.
- # export HADOOP_PID_DIR=/var/hadoop/pids
-
- # A string representing this instance of hadoop. $USER by default.
- # export HADOOP_IDENT_STRING=$USER
-
- # The scheduling priority for daemon processes. See 'man nice'.
- # export HADOOP_NICENESS=10
这里主要配置fs.default.name(用来指定namenode)和hadoop.tmp.dir(默认的hdfs的tmp目录位置),这里可以不设置hadoop.tmp.dir,那么就会保存在默认的/tmp下,每次重启机器都会丢失数据。
点击(此处)折叠或打开
- <?xml version="1.0"?>
- <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-
- <!-- Put site-specific property overrides in this file. -->
-
- <configuration>
- <property>
- <name>fs.default.name</name>
- <value>hdfs://localhost:9000</value>
- </property>
- <property>
- <name>hadoop.tmp.dir</name>
- <value>/home/hadoop/hdfs/tmp</value>
- </property>
-
- </configuration>
这里的dfs.replication用来设置每份数据块的副本数目,默认是3,因为我们是在单机上配置的伪分布模式,因此设为1。dfs.name.dir和dfs.data.dir非常重要,用来设置存放hdfs中namenode和datanode数据的本地存放位置。这里如果设置不好,后续会出现多个错误。当然你也可以不设置采用默认的/tmp下的目录,但是同样重启会丢失数据。
点击(此处)折叠或打开
- <?xml version="1.0"?>
- <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-
- <!-- Put site-specific property overrides in this file. -->
-
- <configuration>
- <property>
- <name>dfs.replication</name>
- <value>1</value>
- </property>
- <property>
- <name>dfs.name.dir</name>
- <value>/home/hadoop/hdfs/name</value>
- </property>
- <property>
- <name>dfs.data.dir</name>
- <value>/home/hadoop/hdfs/data</value>
- </property>
-
- </configuration>
点击(此处)折叠或打开
- <?xml version="1.0"?>
- <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-
- <!-- Put site-specific property overrides in this file. -->
-
- <configuration>
- <property>
- <name>mapred.job.tracker</name>
- <value>localhost:9001</value>
- </property>
-
- </configuration>
![](https://i-blog.csdnimg.cn/blog_migrate/096289eccd36d7754986dac11ef7497a.png)
提示我们存储的HDFS目录要么不存在要么没有权限:
FSNamesystem initialization failed.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:Directory /home/hadoop/hdfs/name is in an inconsistent state: storage directory does not exist or is not accessible.
查看发现已经生成了hdfs目录,那么问题就是权限了,将/home下的hadoop目录权限由755设为775,然后重新运行hadoop,成功:
![](https://i-blog.csdnimg.cn/blog_migrate/5b6c66a351407b3ee54b99e26ca0e4ac.png)
在进行安装hbase之前,我们先来按照官方的方法测试一下伪分布式的hadoop,看看安装是否成功:首先将conf下的所有文件拷贝到hdfs上的input目录中,然后运行jar文件将结果存储到hdfs中的output目录中,最后从output目录中查看结果:
![](https://i-blog.csdnimg.cn/blog_migrate/dab1502e18d08322bb1986f75f86ba9e.png)
点击(此处)折叠或打开
- $ bin/hadoop fs -put conf input
-
- $ bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'
-
- Copy the output files from the distributed filesystem to the local filesytem and examine them:
- $ bin/hadoop fs -get output output
- $ cat output/*
-
- or
-
- View the output files on the distributed filesystem:
- $ bin/hadoop fs -cat output/*
-
- When you're done, stop the daemons with:
- $ bin/stop-all.sh