hadoop 单机节点集群安装

最新推荐文章于 2021-10-27 23:58:13 发布

book_yjq8116

最新推荐文章于 2021-10-27 23:58:13 发布

阅读量163

点赞数

文章标签： hadoop

本文链接：https://blog.csdn.net/book_yjq8116/article/details/84341909

版权

安装 ubantun 11.10 (乌班图)，root：hadoop/hadoop

安装yum：sudo apt-get install yum

更改 root密码：sudo passwd root 输入两次密码即可

安装ssh：apt-get install ssh

配置ssh免密码登录

root@ubuntu:~# ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
Generating public/private dsa key pair.
Created directory '/root/.ssh'.
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
be:03:05:4b:ea:dd:5e:c0:43:e7:a0:ce:13:26:5b:32 root@ubuntu
The key's randomart image is:
+--[ DSA 1024]----+
| |
| o o . |
| o * + |
| E * = . |
| . @ +So |
| o *.. . |
| +.. |
| o. |
| .. |
+-----------------+

root@ubuntu:~# cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
root@ubuntu:~# cd .ssh/
root@ubuntu:~/.ssh# ls
authorized_keys id_dsa id_dsa.pub
root@ubuntu:~/.ssh# cat authorized_keys
ssh-dss AAAAB3NzaC1kc3MAAACBANeaZTGYfT+t4C2EERqIOx6E7KOvrz/+kDSgBSMLz0rCKdXVMdVsfAmqXumfnq7StnFM2WYnz11tmGvHymmJQNrVvd4pAUFKd/oX6uCNpjXA45CMCivXNq67dBM1SDhhBzpntxLNDpU1GVniegiD2a6gFREzhdKe9a2OOA6xoECLAAAAFQDhsAFK1H0VezETQoXe3aWYnm9QRwAAAIBmE8x14azfEhaO59rF7uVlQbjKIOMcBtN9Q0tpuLNkX9bCaqVNMsTanaDXJKoW/cwboRY/32HET/Noc7uqfJZk4YY87ihaPsEeoOZ8uZpknWKDul6j3sbk1yroor16wqCA8CK2Z1Ro22lRY+WHtdeEFfFZd6vvlNxBwQTQCbKuggAAAIAB0gp4uSAY6DEiVnxLfjDx2fEqggGjIFnjfu8PoKV8tbtWpdoayYaw758gN3r3UptOWD/p9//MqVP4CYWwADi1jyTFduRJMjQX91e3VcxH2T69t0c5CJQIZEdleeW2pI1FYN3GNUoj2PD193zWINJTBdQhvW0WWs/2n2XPmqcNEw== root@ubuntu
root@ubuntu:~/.ssh# ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is 0e:1d:c9:40:67:53:db:fb:51:1b:a8:ce:35:32:29:3d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 11.10 (GNU/Linux 3.0.0-12-generic i686)

* Documentation: https://help.ubuntu.com/

New release '12.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

安装rsync：apt-get install rsync

安装vsftpd：apt-get install vsftpd

启动vsftpd：service vsftpd start/stop

解压缩gz：gzip -d hadoop-1.1.0.tar.gz

解压缩tar：tar -xf hadoop-1.1.0.tar

设置JDK：
vi .profile
export JAVA_HOME=/jdk1.6
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

设置hadoop：
修改conf/hadoop-env.sh：export JAVA_HOME=jdk1.6
修改hadoop核心配置文件core-site.xml ，这里配置的是HDFS的地址和端口：
vi conf/core-site.xml


<configuration>
<property> 
      <name>fs.default.name</name> 
       <value>hdfs://localhost:9000</value>  
</property>
<property> 
       <name>dfs.replication</name> 
       <value>1</value>  
</property>
<property>  
       <name>hadoop.tmp.dir</name> 
       <value>/home/hadoop/tmp</value>  
</property>
</configuration>

修改hadoop中HDFS的配置hdfs-site.xml，配置的备份方式默认为3，因为安装的是单机版，所以修改为1

vi conf/hdfs-site.xml

<configuration>
<property>
     <name>dfs.replication</name>
     <value>1</value>
</property>
</configuration>

修改hadoop中MapReduce的配置文件maprd-site.xml，配置的是JobTracker的地址和端口：
vi conf/maprd-site.xml

<configuration>
<property>
     <name>mapred.job.tracker</name>
     <value>localhost:9001</value>
</property>
</configuration>

启动hadoop，在启动之前，需要格式化hadoop的文件系统HDFS，进度hadoop文件夹，输入下面命令：
bin/hadoop namenode -format

root@ubuntu:/hadoop-1.1.0/bin# ./hadoop namenode -format
12/10/29 18:54:58 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ubuntu/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.1.0
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1394289; compiled by 'hortonfo' on Thu Oct 4 22:06:49 UTC 2012
************************************************************/
12/10/29 18:54:58 INFO util.GSet: VM type = 32-bit
12/10/29 18:54:58 INFO util.GSet: 2% max memory = 19.33375 MB
12/10/29 18:54:58 INFO util.GSet: capacity = 2^22 = 4194304 entries
12/10/29 18:54:58 INFO util.GSet: recommended=4194304, actual=4194304
12/10/29 18:54:59 INFO namenode.FSNamesystem: fsOwner=root
12/10/29 18:54:59 INFO namenode.FSNamesystem: supergroup=supergroup
12/10/29 18:54:59 INFO namenode.FSNamesystem: isPermissionEnabled=true
12/10/29 18:54:59 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
12/10/29 18:54:59 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
12/10/29 18:54:59 INFO namenode.NameNode: Caching file names occuring more than 10 times
12/10/29 18:54:59 INFO common.Storage: Image file of size 110 saved in 0 seconds.
12/10/29 18:55:00 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/tmp/hadoop-root/dfs/name/current/edits
12/10/29 18:55:00 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/tmp/hadoop-root/dfs/name/current/edits
12/10/29 18:55:00 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.
12/10/29 18:55:00 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1
************************************************************/
root@ubuntu:/hadoop-1.1.0/bin#

启动hadoop，输入 bin/start-all.sh，这个命令启动了所有服务。

root@ubuntu:/hadoop-1.1.0/bin# ./start-all.sh
starting namenode, logging to /hadoop-1.1.0/libexec/../logs/hadoop-root-namenode-ubuntu.out
localhost: starting datanode, logging to /hadoop-1.1.0/libexec/../logs/hadoop-root-datanode-ubuntu.out
localhost: starting secondarynamenode, logging to /hadoop-1.1.0/libexec/../logs/hadoop-root-secondarynamenode-ubuntu.out
starting jobtracker, logging to /hadoop-1.1.0/libexec/../logs/hadoop-root-jobtracker-ubuntu.out
localhost: starting tasktracker, logging to /hadoop-1.1.0/libexec/../logs/hadoop-root-tasktracker-ubuntu.out

此语句执行后会列出已启动的东西NameNode,JobTracker,SecondaryNameNode...如果NameNode没有成功启动的话就要先执行"bin/stop-all.sh"停掉所有东西，然后重新格式化namenode，再启动

验证hadoop是否安装成功，访问：
http://localhost:50030 （MapReduce的web界面）
http://localhost:50070 （这个是 HDFS的web界面）

root@ubuntu:/jdk1.6/bin# ./jps
11325 JobTracker
13345 Jps
10826 NameNode
11036 DataNode
11541 TaskTracker
11252 SecondaryNameNode

Hadoop 从三个角度将主机划分为两种角色
1、Master 和 slave
2、从HDFS角度，主机分为：namenode 和 datanode（在分布式系统中，目录的管理是关键，管理目录的就相当于主任，而namenode就是目录管理者）
3、从MapReduce角度，主机分为：JobTracker 和 Task Tracker（一个job经常被划分为多个task，从这个角度不难理解他们之间的关系）

【跑 wordcount】

1、准备需要进行wordcount的文件
vi /tmp/test.txt
(打开后随便输入一些内容，如"mu ha ha ni da ye da ye da"，然后保存退出）

2、将准备的测试文件上传到dfs文件系统中的firstTest目录下
hadoop dfs -copyFromLocal /tmp/test.txt firstTest
（注：如dfs中不包含firstTest目录的话就会自动创建一个，关于查看dfs文件系统中已有目录的指令为"hadoop dfs -ls")

3、执行wordcount
hadoop jar hadoop-mapred-example0.21.0.jar wordcount firstTest result
（注：此语句意为“对firstTest下的所有文件执行wordcount，将统计结果输出到result文件夹中”，若result文件夹不存在则会自动创建一个）

hadoop-mapred-example0.21.0.jar 在 hadoop的根目录下

root@ubuntu:/hadoop-1.1.0/bin# ./hadoop jar ../hadoop-examples-1.1.0.jar wordcount firstTest result
12/10/29 19:24:32 INFO input.FileInputFormat: Total input paths to process : 1
12/10/29 19:24:32 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/10/29 19:24:32 WARN snappy.LoadSnappy: Snappy native library not loaded
12/10/29 19:24:33 INFO mapred.JobClient: Running job: job_201210291856_0001
12/10/29 19:24:34 INFO mapred.JobClient: map 0% reduce 0%
12/10/29 19:24:52 INFO mapred.JobClient: map 100% reduce 0%
12/10/29 19:25:03 INFO mapred.JobClient: map 100% reduce 100%
12/10/29 19:25:04 INFO mapred.JobClient: Job complete: job_201210291856_0001
12/10/29 19:25:04 INFO mapred.JobClient: Counters: 29
12/10/29 19:25:04 INFO mapred.JobClient: Job Counters
12/10/29 19:25:04 INFO mapred.JobClient: Launched reduce tasks=1
12/10/29 19:25:04 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=16020
12/10/29 19:25:04 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/10/29 19:25:04 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/10/29 19:25:04 INFO mapred.JobClient: Launched map tasks=1
12/10/29 19:25:04 INFO mapred.JobClient: Data-local map tasks=1
12/10/29 19:25:04 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=11306
12/10/29 19:25:04 INFO mapred.JobClient: File Output Format Counters
12/10/29 19:25:04 INFO mapred.JobClient: Bytes Written=26
12/10/29 19:25:04 INFO mapred.JobClient: FileSystemCounters
12/10/29 19:25:04 INFO mapred.JobClient: FILE_BYTES_READ=52
12/10/29 19:25:04 INFO mapred.JobClient: HDFS_BYTES_READ=134
12/10/29 19:25:04 INFO mapred.JobClient: FILE_BYTES_WRITTEN=47699
12/10/29 19:25:04 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=26
12/10/29 19:25:04 INFO mapred.JobClient: File Input Format Counters
12/10/29 19:25:04 INFO mapred.JobClient: Bytes Read=28
12/10/29 19:25:04 INFO mapred.JobClient: Map-Reduce Framework
12/10/29 19:25:04 INFO mapred.JobClient: Map output materialized bytes=52
12/10/29 19:25:04 INFO mapred.JobClient: Map input records=1
12/10/29 19:25:04 INFO mapred.JobClient: Reduce shuffle bytes=52
12/10/29 19:25:04 INFO mapred.JobClient: Spilled Records=10
12/10/29 19:25:04 INFO mapred.JobClient: Map output bytes=64
12/10/29 19:25:04 INFO mapred.JobClient: CPU time spent (ms)=6830
12/10/29 19:25:04 INFO mapred.JobClient: Total committed heap usage (bytes)=210698240
12/10/29 19:25:04 INFO mapred.JobClient: Combine input records=9
12/10/29 19:25:04 INFO mapred.JobClient: SPLIT_RAW_BYTES=106
12/10/29 19:25:04 INFO mapred.JobClient: Reduce input records=5
12/10/29 19:25:04 INFO mapred.JobClient: Reduce input groups=5
12/10/29 19:25:04 INFO mapred.JobClient: Combine output records=5
12/10/29 19:25:04 INFO mapred.JobClient: Physical memory (bytes) snapshot=181235712
12/10/29 19:25:04 INFO mapred.JobClient: Reduce output records=5
12/10/29 19:25:04 INFO mapred.JobClient: Virtual memory (bytes) snapshot=751153152
12/10/29 19:25:04 INFO mapred.JobClient: Map output records=9

4、查看结果
hadoop dfs -cat result/part-r-00000
（注：结果文件默认是输出到一个名为“part-r-*****”的文件中的，可用指令“hadoop dfs -ls result”查看result目录下包含哪些文件）