Hadoop: Setting up a Single Node Cluster
- [first time] install ssh, rsync
注意: 修改$HADOOP_HOME/etc/hadoop/hadoop-env.sh
中的JAVA_HOME。这一步很重要,然后启动时会报错。
Unpack the downloaded Hadoop distribution. In the distribution, edit the file etc/hadoop/hadoop-env.sh to define some parameters as follows:
# set to the root of your Java installation
export JAVA_HOME=/usr/java/latest
Standalone operation
By default, Hadoop is configured to run in a non-distributed mode, as a single Java process. This is useful for debugging.Execution
The following instructions are to run a MapReduce job locally. If you want to execute a job on YARN, see YARN on Single Node.Format the filesystem:
$ bin/hdfs namenode -format
Start NameNode daemon and DataNode daemon:
$ sbin/start-dfs.sh
The hadoop daemon log output is written to the HADOOPLOGDIRdirectory(defaultsto HADOOP_HOME/logs).
Browse the web interface for the NameNode; by default it is available at:
NameNode - http://localhost:50070/Make the HDFS directories required to execute MapReduce jobs:
bin/hdfsdfs−mkdir/user bin/hdfs dfs -mkdir /user/
Copy the input files into the distributed filesystem:
$ bin/hdfs dfs -put etc/hadoop input
Run some of the examples provided:
注意:跑这个例子时要先确认当前目录底下没有output目录,否则执行这个任务会报错。因为hadoop不会覆盖这个目录。$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar grep input output ‘dfs[a-z.]+’
Examine the output files: Copy the output files from the distributed filesystem to the local filesystem and examine them:
bin/hdfsdfs−getoutputoutput cat output/*
or
View the output files on the distributed filesystem:
$ bin/hdfs dfs -cat output/*
When you’re done, stop the daemons with:
$ sbin/stop-dfs.sh
参考http://www.powerxing.com/install-hadoop/
常见问题:
执行hadoop namenode -format之前必须删除缓存文件,不然会报错,导致找不到datanode
默认的缓存在/tmp 目录下。