hadoop0.20.2完全分布模式安装和配置

最新推荐文章于 2019-11-20 11:19:42 发布
最新推荐文章于 2019-11-20 11:19:42 发布
阅读量115 收藏
点赞数

-----------------------------------------------------------
hadoop集群规划
IP地址          hostname
------------    --------
10.10.10.100 master(namenode,secondary namenode,job tracker)
10.10.10.101 slave1(datanode,tasktracker)
10.10.10.102 slave2(datanode,tasktracker)
-----------------------------------------------------------

虚拟机软件VMWare Server2.0
操作系统：RedHat Enterprise Linux Server 5.3(32bit)*3
hadoop版本：0.20.2
jdk版本：1.7
注意：各操作用户和所在主机名请注意查看命令行的提示符
-----------------------------------------------------------

1、编辑/etc/hosts(三台机器都操作)
127.0.0.1       localhost
10.10.10.100    master
10.10.10.101    slave1
10.10.10.102    slave2

2、创建hadoop用户组和用户(三台机器都操作)
[root@master ~]# groupadd hadoop
[root@master ~]# useradd hadoop
[root@master ~]# passwd hadoop
Changing password for user hadoop.
New UNIX password:
BAD PASSWORD: it is based on a dictionary word
Retype new UNIX password:
passwd: all authentication tokens updated successfully.

3、上传需要用到的软件，这里用smb上传到hadoop家目录的software目录下(三台机器都操作)
[hadoop@master software]$ pwd
/home/hadoop/software
[hadoop@master software]$ ll
total 162096
-rwxr--r-- 1 hadoop hadoop  44575568 Feb  3 15:24 hadoop-0.20.2.tar.gz
-rwxr--r-- 1 hadoop hadoop 121236291 Jan  3 11:15 jdk-7u45-linux-i586.rpm

4、安装jdk(三台机器都操作)
[root@master ~]# cd /home/hadoop/software/
[root@master software]# ll
total 162096
-rwxr--r-- 1 hadoop hadoop  44575568 Feb  3 15:24 hadoop-0.20.2.tar.gz
-rwxr--r-- 1 hadoop hadoop 121236291 Jan  3 11:15 jdk-7u45-linux-i586.rpm
[root@master software]# rpm -ivh jdk-7u45-linux-i586.rpm
Preparing...                ########################################### [100%]
   1:jdk                    ########################################### [100%]
Unpacking JAR files...
        rt.jar...
        jsse.jar...
        charsets.jar...
        tools.jar...
        localedata.jar...
        jfxrt.jar...
        plugin.jar...
        javaws.jar...
        deploy.jar...
[root@master software]# vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.7.0_45
export PATH=$JAVA_HOME/bin/:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

[root@master software]# source /etc/profile
[root@master software]# java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) Client VM (build 24.45-b08, mixed mode, sharing)

5、配置hadoop用户能够无密码登陆
--master节点
[hadoop@master ~]$ chmod -R 755 /home/hadoop
[hadoop@master ~]$ mkdir ~/.ssh
[hadoop@master ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
b4:be:4c:c0:b8:2c:bf:8f:ae:5c:8d:8b:4d:b8:90:87 hadoop@master
[hadoop@master ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_dsa.
Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.
The key fingerprint is:
da:02:17:e3:10:42:38:a5:e9:12:f2:36:e5:22:14:6d hadoop@master

--slave1节点
[hadoop@slave1 ~]$ chmod -R 755 /home/hadoop
[hadoop@slave1 ~]$ mkdir ~/.ssh
[hadoop@slave1 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
26:d6:66:43:36:ba:69:9f:95:d2:bb:4e:80:80:2f:0f hadoop@slave1
[hadoop@slave1 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_dsa.
Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.
The key fingerprint is:
15:67:b5:ec:37:c4:db:ef:34:25:6c:8d:49:40:5d:2c hadoop@slave1

--slave2节点

[hadoop@slave2 ~]$ chmod -R 755 /home/hadoop
[hadoop@slave2 ~]$ mkdir ~/.ssh
[hadoop@slave2 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
b8:03:99:20:ec:0e:49:af:07:1c:66:15:7c:66:2e:03 hadoop@slave2
[hadoop@slave2 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_dsa.
Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.
The key fingerprint is:
bc:52:59:f7:08:9a:37:17:57:af:dd:45:61:43:1e:e0 hadoop@slave2

--master节点
[hadoop@master ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@master ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

[hadoop@master ~]$ ssh slave1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'slave1 (10.10.10.101)' can't be established.
RSA key fingerprint is 99:ef:c2:9e:28:e3:b6:83:e2:00:eb:a3:ee:ad:29:d8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave1,10.10.10.101' (RSA) to the list of known hosts.
hadoop@slave1's password:
[hadoop@master ~]$ ssh slave1 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
hadoop@slave1's password:

[hadoop@master ~]$ ssh slave2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'slave2 (10.10.10.102)' can't be established.
RSA key fingerprint is 99:ef:c2:9e:28:e3:b6:83:e2:00:eb:a3:ee:ad:29:d8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave2,10.10.10.102' (RSA) to the list of known hosts.
hadoop@slave2's password:
[hadoop@master ~]$ ssh slave2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
hadoop@slave2's password:

[hadoop@master ~]$ scp ~/.ssh/authorized_keys slave1:~/.ssh/authorized_keys
hadoop@slave1's password:
authorized_keys                                                                                   100% 2994     2.9KB/s   00:00   
[hadoop@master ~]$ scp ~/.ssh/authorized_keys slave2:~/.ssh/authorized_keys
hadoop@slave2's password:
authorized_keys                                                                                   100% 2994     2.9KB/s   00:00
--分别修改下各个节点的.ssh目录权限

[hadoop@master ~]$ chmod -R 700 ~/.ssh
[hadoop@slave1 ~]$ chmod -R 700 ~/.ssh
[hadoop@slave2 ~]$ chmod -R 700 ~/.ssh

--在master上进行测试
[hadoop@master ~]$ ssh master date
The authenticity of host 'master (10.10.10.100)' can't be established.
RSA key fingerprint is 99:ef:c2:9e:28:e3:b6:83:e2:00:eb:a3:ee:ad:29:d8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,10.10.10.100' (RSA) to the list of known hosts.
Fri Feb  7 00:10:42 CST 2014
[hadoop@master ~]$ ssh master date
Fri Feb  7 00:10:45 CST 2014
[hadoop@master ~]$ ssh slave1 date
Fri Feb  7 00:09:46 CST 2014
[hadoop@master ~]$ ssh slave2 date
Fri Feb  7 00:10:52 CST 2014
--在slave1上进行测试

[hadoop@slave1 ~]$ ssh master date
The authenticity of host 'master (10.10.10.100)' can't be established.
RSA key fingerprint is 99:ef:c2:9e:28:e3:b6:83:e2:00:eb:a3:ee:ad:29:d8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,10.10.10.100' (RSA) to the list of known hosts.
Fri Feb  7 00:11:30 CST 2014
[hadoop@slave1 ~]$ ssh master date
Fri Feb  7 00:11:32 CST 2014
[hadoop@slave1 ~]$ ssh slave1 date
The authenticity of host 'slave1 (10.10.10.101)' can't be established.
RSA key fingerprint is 99:ef:c2:9e:28:e3:b6:83:e2:00:eb:a3:ee:ad:29:d8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave1,10.10.10.101' (RSA) to the list of known hosts.
Fri Feb  7 00:10:34 CST 2014
[hadoop@slave1 ~]$ ssh slave1 date
Fri Feb  7 00:10:36 CST 2014
[hadoop@slave1 ~]$ ssh slave2 date
The authenticity of host 'slave2 (10.10.10.102)' can't be established.
RSA key fingerprint is 99:ef:c2:9e:28:e3:b6:83:e2:00:eb:a3:ee:ad:29:d8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave2,10.10.10.102' (RSA) to the list of known hosts.
Fri Feb  7 00:11:42 CST 2014
[hadoop@slave1 ~]$ ssh slave2 date
Fri Feb  7 00:11:44 CST 2014
--在slave2上进行测试

[hadoop@slave2 ~]$ ssh master date
The authenticity of host 'master (10.10.10.100)' can't be established.
RSA key fingerprint is 99:ef:c2:9e:28:e3:b6:83:e2:00:eb:a3:ee:ad:29:d8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,10.10.10.100' (RSA) to the list of known hosts.
Fri Feb  7 00:12:13 CST 2014
[hadoop@slave2 ~]$ ssh master date
Fri Feb  7 00:12:15 CST 2014
[hadoop@slave2 ~]$ ssh slave1  date
The authenticity of host 'slave1 (10.10.10.101)' can't be established.
RSA key fingerprint is 99:ef:c2:9e:28:e3:b6:83:e2:00:eb:a3:ee:ad:29:d8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave1,10.10.10.101' (RSA) to the list of known hosts.
Fri Feb  7 00:11:17 CST 2014
[hadoop@slave2 ~]$ ssh slave1  date
Fri Feb  7 00:11:18 CST 2014
[hadoop@slave2 ~]$ ssh slave2  date
The authenticity of host 'slave2 (10.10.10.102)' can't be established.
RSA key fingerprint is 99:ef:c2:9e:28:e3:b6:83:e2:00:eb:a3:ee:ad:29:d8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave2,10.10.10.102' (RSA) to the list of known hosts.
Fri Feb  7 00:12:26 CST 2014
[hadoop@slave2 ~]$ ssh slave2  date
Fri Feb  7 00:12:28 CST 2014

6、在master上安装hadoop
[hadoop@master software]$ pwd
/home/hadoop/software
[hadoop@master software]$ ll
total 162096
-rwxr-xr-x 1 hadoop hadoop  44575568 Feb  3 15:24 hadoop-0.20.2.tar.gz
-rwxr-xr-x 1 hadoop hadoop 121236291 Jan  3 11:15 jdk-7u45-linux-i586.rpm
[hadoop@master software]$ tar -zxvf hadoop-0.20.2.tar.gz
[hadoop@master software]$ ll
total 162100
drwxr-xr-x 12 hadoop hadoop      4096 Feb 19  2010 hadoop-0.20.2
-rwxr-xr-x  1 hadoop hadoop  44575568 Feb  3 15:24 hadoop-0.20.2.tar.gz
-rwxr-xr-x  1 hadoop hadoop 121236291 Jan  3 11:15 jdk-7u45-linux-i586.rpm
[hadoop@master software]$ mv hadoop-0.20.2 /home/hadoop
[hadoop@master software]$ cd
[hadoop@master ~]$ ll
total 8
drwxr-xr-x 12 hadoop hadoop 4096 Feb 19  2010 hadoop-0.20.2
drwxr-xr-x  2 hadoop hadoop 4096 Feb  3 23:59 software


7、在master上配置hadoop
[hadoop@master conf]$ pwd
/home/hadoop/hadoop-0.20.2/conf
[hadoop@master conf]$ ll
total 56
-rw-rw-r-- 1 hadoop hadoop 3936 Feb 19  2010 capacity-scheduler.xml
-rw-rw-r-- 1 hadoop hadoop  535 Feb 19  2010 configuration.xsl
-rw-rw-r-- 1 hadoop hadoop  178 Feb 19  2010 core-site.xml
-rw-rw-r-- 1 hadoop hadoop 2237 Feb 19  2010 hadoop-env.sh
-rw-rw-r-- 1 hadoop hadoop 1245 Feb 19  2010 hadoop-metrics.properties
-rw-rw-r-- 1 hadoop hadoop 4190 Feb 19  2010 hadoop-policy.xml
-rw-rw-r-- 1 hadoop hadoop  178 Feb 19  2010 hdfs-site.xml
-rw-rw-r-- 1 hadoop hadoop 2815 Feb 19  2010 log4j.properties
-rw-rw-r-- 1 hadoop hadoop  178 Feb 19  2010 mapred-site.xml
-rw-rw-r-- 1 hadoop hadoop   10 Feb 19  2010 masters
-rw-rw-r-- 1 hadoop hadoop   10 Feb 19  2010 slaves
-rw-rw-r-- 1 hadoop hadoop 1243 Feb 19  2010 ssl-client.xml.example
-rw-rw-r-- 1 hadoop hadoop 1195 Feb 19  2010 ssl-server.xml.example

[hadoop@master conf]$ vi hadoop-env.sh
# The java implementation to use.  Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
export JAVA_HOME=/usr/java/jdk1.7.0_45

[hadoop@master conf]$ vi core-site.xml


fs.default.name
hdfs://master:9000

[hadoop@master conf]$ vi hdfs-site.xml


dfs.data.dir
/home/hadoop/hadoop-data


dfs.name.dir
/home/hadoop/hadoop-name


fs.checkpoint.dir
/home/hadoop/hadoop-namesecondary


dfs.replication
2

[hadoop@master conf]$ vi mapred-site.xml


mapred.job.tracker
master:9001

[hadoop@master conf]$ vi masters
master

[hadoop@master conf]$ vi slaves
slave1
slave2

8、分发master上配置好的的hadoop软件到slave1和slave2节点
[hadoop@master ~]$ scp -r hadoop-0.20.2/ slave1:~/
[hadoop@master ~]$ scp -r hadoop-0.20.2/ slave2:~/

9、格式化hdfs（在master节点上）
[hadoop@master bin]$ pwd
/home/hadoop/hadoop-0.20.2/bin
[hadoop@master bin]$ ll
total 64
-rwxr-xr-x 1 hadoop hadoop 9998 Feb 19  2010 hadoop
-rwxr-xr-x 1 hadoop hadoop 1966 Feb 19  2010 hadoop-config.sh
-rwxr-xr-x 1 hadoop hadoop 3690 Feb 19  2010 hadoop-daemon.sh
-rwxr-xr-x 1 hadoop hadoop 1227 Feb 19  2010 hadoop-daemons.sh
-rwxr-xr-x 1 hadoop hadoop 2710 Feb 19  2010 rcc
-rwxr-xr-x 1 hadoop hadoop 2043 Feb 19  2010 slaves.sh
-rwxr-xr-x 1 hadoop hadoop 1066 Feb 19  2010 start-all.sh
-rwxr-xr-x 1 hadoop hadoop  965 Feb 19  2010 start-balancer.sh
-rwxr-xr-x 1 hadoop hadoop 1645 Feb 19  2010 start-dfs.sh
-rwxr-xr-x 1 hadoop hadoop 1159 Feb 19  2010 start-mapred.sh
-rwxr-xr-x 1 hadoop hadoop 1019 Feb 19  2010 stop-all.sh
-rwxr-xr-x 1 hadoop hadoop 1016 Feb 19  2010 stop-balancer.sh
-rwxr-xr-x 1 hadoop hadoop 1146 Feb 19  2010 stop-dfs.sh
-rwxr-xr-x 1 hadoop hadoop 1068 Feb 19  2010 stop-mapred.sh
[hadoop@master bin]$ ./hadoop namenode -format

14/02/07 00:46:15 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = master/10.10.10.100
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
14/02/07 00:46:15 INFO namenode.FSNamesystem: fsOwner=hadoop,hadoop
14/02/07 00:46:15 INFO namenode.FSNamesystem: supergroup=supergroup
14/02/07 00:46:15 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/02/07 00:46:15 INFO common.Storage: Image file of size 96 saved in 0 seconds.
14/02/07 00:46:15 INFO common.Storage: Storage directory /home/hadoop/hadoop-name has been successfully formatted.
14/02/07 00:46:15 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/10.10.10.100
************************************************************/


10、启动hadoop
[hadoop@master bin]$ ./start-all.sh （在master节点上）
starting namenode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-namenode-master.out
slave2: starting datanode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-datanode-slave2.out
slave1: starting datanode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-datanode-slave1.out
master: starting secondarynamenode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-secondarynamenode-master.out
starting jobtracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-jobtracker-master.out
slave1: starting tasktracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-tasktracker-slave1.out
slave2: starting tasktracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-tasktracker-slave2.out


11、查看hadoop进程
[hadoop@master bin]$ jps
8131 SecondaryNameNode
8201 JobTracker
8269 Jps
7968 NameNode

[hadoop@slave1 logs]$ jps
7570 DataNode
7722 Jps
7675 TaskTracker

[hadoop@slave2 conf]$ jps
7667 TaskTracker
7562 DataNode
7714 Jps


12、访问http服务
http://10.10.10.100:50030/（jobtracker的HTTP服务器地址和端口）
http://10.10.10.100:50060/（taskertracker的HTTP服务器地址和端口）
http://10.10.10.100:50070/（namenode的HTTP服务器地址和端口）
http://10.10.10.100:50075/（datanode的HTTP服务器地址和端口）
http://10.10.10.100:50090/（secondary namenode的HTTP服务器地址和端口）

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/26613085/viewspace-1077424/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/26613085/viewspace-1077424/

cuiba1921
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
hadoop0.20.2配置手册
10-11
hadoop0.20.2版本在虚拟机中的安装指南，经过测试可以实现
Hadoop完全分布模式安装.pdf
04-29
Hadoop完全分布模式安装
Hadoop入门之安装配置(hadoop-0.20.2)
weixin_30611509的博客
10-27 105
Hadoop，简单理解为HDFS(分布式存储)+Mapreduce(分布式处理)，专为离线和大规模数据分析而设计。 Hadoop可以把很多linux的廉价PC组成分布式结点，然后编程人员也不需要知道分布式算法之类的，只需根据mapreduce的规则定义好的接口方法，剩下的就交给hadoop，它会自动把相关的计算分布到各个结点上去，然后得到结果。 比如，有一个...
Hadoop0.20.2的安装与配置
thomas0988
12-19 119
本文主要介绍在 Ubuntu 环境下 Hadoop0.20.2 的安装及配置： 步聚一、机器分配及用途 hdfs-m--------------172.16.11.197   (Master--NameNode) hdfs-s1-------------172.16.11.149   (Slave1--DataNode) hdfs-s2-------------172.16.11....
hadoop 三个节点集群的安装部署
congji3615的博客
11-22 181
MicrosoftInternetExplorer402DocumentNotSpecified7.8Normal0实验环境（三台机）一台做master（存放namenode、jobtracker）另外两台做slave（存...
centos7下调试集群三台机器实现免密登陆--hadoop安装系列之一
wangli的博客
04-02 2881
一、实验目标 安装三台centos7虚拟机，组建hadoop实验集群，centos是从centos7官网下载的最新版本，默认系统安装，创建hadoop用户组，新建用户hadoop并加入hadoop组。 二、实验环境介绍 三台机器的网络主机配置如下： 192.168.10.166 master 192.168.10.167 slave01 192.168.10.168 slave02
hadoop安装及其注意事项
sunrui888的博客
08-20 1995
安装hadoop步骤： 一linux系统配置 两台主机都要配置 root用户 1）配置时钟同步 2）配置主机名 3）使用setup配置网络环境（ip地址关闭防火墙） 4）关闭防火墙 5）配置hosts列表（用主机名代替ip 建立主机名与ip一一对应的关系） 6）安装jdk(apache要用jdk hadoop基于apache) 7）免密钥登陆（一台服务器一个客户端） 二 hado
hadoop_完全分布式模式安装和配置
04-09
虚拟机实现的hadoop完全分布式模式的安装和配置，有详细的操作步骤和截图，亲测成功。
hadoop-0.20.2安装配置
07-21
hadoop-0.20.2的安装配置，其他环境为JDK1.7.0_25,Ubuntu-12.04
Ubuntu Hadoop 0.20.2 安装配置说明
08-20
本文档适合初学者，指引初学者或初次接触 Hadoop 的人成功安装 Hadoop ，使用代码执行简单的 HDFS 操作，并利用 MapReduce 执行分布式计算示例代码。其中 MapReduce 示例代码为 Hadoop 官方带的 hadoop-0.20.2-...
hadoop学习笔记之-hbase完全分布模…
weixin_30570101的博客
03-24 204
安装环境： OS: Oracle linux 5.6 JDK: jdk1.6.0_18 Hadoop: hadoop-0.20.2 Hbase: hbase-0.90.5 安装准备： 1. Jdk环境已安装：版本为1.6以上 2. hadoop环境已安装：完全分布模式安装如下 http://blog.csdn.net/lichangzai/article/details/82068...
hadoop-0.20.2开发环境搭建
leaderYU的专栏
03-12 1736
折腾了蛮久，参看了网上很多博客文章，我的hadoop环境终于搭建起来。现在回想起来，好像也没有什么难的，不过一步步照着来就是了。但是最初搭建的时候确实遇到了很多问题。现在就自己能记得的，把重要的步骤记录一下。 准备工作：hadoop-0.20.2、jdk-7u45-linux-x64、VMware-workstation-full-v10.0.0-1295980、ubuntu-12.04-des
hadoop常用shell
Zonzereal的博客
11-06 3172
Shell操作(常用操作) 1、 ls 用法1：hadoop fs -ls / 功能：列出hdfs文件系统根目录下的目录和文件 用法2：hadoop fs -ls -R / 功能：列出hdfs文件系统所有的目录和文件 2、 put 用法1：hadoop fs -put 功能：hdfs file的父目录一定要存在，否则命令不会执行 用法2：hadoop fs -put 功
漫谈Hadoop HDFS Balancer
weixin_34232744的博客
02-16 296
Hadoop的HDFS集群非常容易出现机器与机器之间磁盘利用率不平衡的情况，比如集群中添加新的数据节点。当HDFS出现不平衡状况的时候，将引发很多问题，比如MR程序无法很好地利用本地计算的优势，机器之间无法达到更好的网络带宽使用率，机器磁盘无法利用等等。可见，保证HDFS中的数据平衡是非常重要的。 在Hadoop中，包含一个Balancer程序，通过运行这个程序，可以使得HDFS集群达到一个平衡...
hadoop Balance 优化经验
breakout_alex的博客
11-20 703
前言 为了不影响业务的正常运行，hdfsbalancer的默认参数都比较小（宽带，并发等），在集群规模比较大，数据不均衡情况比较严重时（例如：上线大量新节点），使用hdfsbalancer无法快速缓解问题，极端情况下，会导致很多节点存储用满，严重影响集群稳定性，且hadoop的一些小版本的balancer都存在一定的bug，本篇文章主要分享下关于hadoopbalancer的一些优化经验...
Hadoop集群日常运维
apple01010105的博客
03-01 670
（一）备份namenode的元数据 namenode中的元数据非常重要，如丢失或者损坏，则整个系统无法使用。因此应该经常对元数据进行备份，最好是异地备份。 1、将元数据复制到远程站点 （1）以下代码将secondary namenode中的元数据复制到一个时间命名的目录下，然后通过scp命令远程发送到其它机器 #!/bin/bash export dirname=/m...
hadoop 启动提示输入password的问题
xiongjiezk的专栏
12-27 7823
hadoop启动时，如果没有配置ssh免秘钥登录的话，会提示要求输入password：[xxx@*** /home/q/www/hadoop-2.6.4]$ sudo -u tomcat sbin/start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh 16/12/27 16:41:36
hadoop HDFS常用文件操作命令
weixin_33901926的博客
04-14 4556
命令基本格式: hadoop fs -cmd < args > 1.ls hadoop fs -ls / 列出hdfs文件系统根目录下的目录和文件 hadoop fs -ls -R / 列出hdfs文件系统所有的目录和文件 2.put hadoop fs -put < local file &...
hadoop进阶---hadoop性能优化(一)---hdfs空间不足的管理优化
直到世界的尽头
03-03 9303
Hadoop 空间不足，hive首先就会没法跑了，进度始终是0%。 将HDFS备份数降低 将默认的备份数3设置为2。 步骤：CDH–&gt;HDFS–&gt;配置–&gt;搜索dfs.replication–&gt;设置为2 删除无用HDFS数据和Hbase表格 主要使用命令hadoop fs -du，hadoop fs -ls，hadoop fs -rm 空间不足根本的解决...
hadoop0.20.2集群安装
最新发布
10-14
好的，关于hadoop0.20.2集群安装，您可以按照以下步骤进行： 1. 下载hadoop0.20.2版本，并解压到指定目录。 2. 配置hadoop-env.sh文件，设置JAVA_HOME和HADOOP_HOME等环境变量。 3. 配置core-site.xml文件，设置...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值