1. Linux虚机换机环境安装
1.1 linux环境安装
1.建议选择虚拟机:VirtualBox
2.Linux版本:Ubuntu
3.安装时选择动态扩展磁盘,最大磁盘容量50G(最大磁盘容量太小,hadoop使用过程中容易出现意想不到的的问题)
4.网络选择桥接网卡(不要选择NAT,不然路由器不会为虚拟机分配独立的IP地址)
5.安装增强功能开启双向开启共享剪切板
6.安装用户使用同样的用户名:如hadoop,在建立分布式环境时需要保证通用的用户名
7.设置hadoop主机名为:hadoop-master hadoop-slave01 hadoop-slave02 (如果忘记设置可以看1.2修改主机名)
ps:具体安装过程不再赘述
安装成功后:ifconfig查看机器ip
机器1的ip
机器2的ip
机器3的ip
1.2 linux主机名修改
- sudo apt-get install vim
- sudo vim /etc/hostname
- 修改为自己想要的域名如:hadoop-master,并保存
- reboot(重启)
- 确认主机名修改成功:hadoop@hadoop-VirtualBox 变为hadoop@hadoop-master
2 ssh和polysh工具安装
ssh和polysh的安装的目的是通过自己的机器去访问虚拟机如:本人macbook去访问virtualbox中的hadoop-master hadoop-slave01 hadoop-slave02 3台虚拟机。
其中ssh需要在四台机器都安装,polysh则只需要在自己本机安装,从而通过一个shell的形式访问3台机器
2.1 ssh安装和使用
- sudo apt-get install ssh (各个虚拟机都要先安装,下面的步骤都可以同ssh远程登录机器操作)
- 在本地机器建立虚拟机host和ip映射关系:/etc/hosts 输入如下内容: 192.168.31.88 hadoop-master
192.168.31.234 hadoop-slave01
192.168.31.186 hadoop-slave02 - ssh hadoop@hadoop-master (输入虚拟机密码)
- ssh-keygen(生成免密登录的几个文件,全部回车即可)
- 切换目标cd .ssh 显示文件ls (id_rsa id_rsa.pub 文件具体意义自己百度)
- 如果要免密登录到一台机器,需要把自己机器的id_rsa.pub文件导入到要登录机器.ssh/authorized_keys文件中
- 到虚拟机上执行命令:echo “ssh-rsa xxxxxxxx” >> .ssh/authorized_keys (”ssh xxx“为本地机器的id_rsa.pub的内容)
- exit关闭虚拟机shell
- ssh hadoop@hadoop-master (不再需要密码)
- 最好是将所有虚拟机的id_rsa.pub也导入到本地机器的authorized_keys中使得互相传文件时不用输密码
- 现在可以无密码的登录三台机器ssh hadoop@hadoop-master ssh hadoop@hadoop-slave01 ssh hadoop@hadoop-slave02
2.2 polysh安装
polysh是批量登录多台机器的工具,如上面如果你要传一个hadoop的安装文件包到虚拟机上,你需要建立三个ssh链接,并在每个ssh的shell中上传同样的文件,而polysh的作用就是你可以在polysh的shell中操作三台机器,如上传文件,只需要一次上传,就在三台虚拟机中都上传了文件。
polysh脚本安装
wget http://guichaz.free.fr/polysh/files/polysh-0.4.tar.gz
tar -zxvf polysh-0.4.tar.gz
cd polysh-0.4
sudo python setup.py install
polysh -help 如果有提示表示安装成功
# 将polysh命令加入path中
echo "export PATH=~/polysh/bin:$PATH" >> ~/.bash_profile
source ~/.bash_profile
polysh使用
echo "alias hadoop-login=\"polysh 'hadoop@hadoop-master' 'hadoop@hadoop-slave<01-02>' \" ">> ~/.bash_profile
source ~/.bash_profile
hadoop-login
列如:hadoop主从机器的host和ip映射
sudo sh -c "echo \"192.168.31.88 hadoop-master\n192.168.31.234 hadoop-slave01\n192.168.31.186 hadoop-slave02\n\" >> /etc/hosts"
cat /etc/hosts
hadoop@hadoop-slave01 : 192.168.31.88 hadoop-master
hadoop@hadoop-slave01 : 192.168.31.234 hadoop-slave01
hadoop@hadoop-slave01 : 192.168.31.186 hadoop-slave02
hadoop@hadoop-master : 192.168.31.88 hadoop-master
hadoop@hadoop-master : 192.168.31.234 hadoop-slave01
hadoop@hadoop-master : 192.168.31.186 hadoop-slave02
hadoop@hadoop-slave02 : 192.168.31.88 hadoop-master
hadoop@hadoop-slave02 : 192.168.31.234 hadoop-slave01
hadoop@hadoop-slave02 : 192.168.31.186 hadoop-slave02
3 hadoop安装
3.1 hadoop安装包下载和上传
- 下载hadoop安装包:hadoop-1.2.1-bin.tar.gz
- scp username@192.168.31.19:/Users/username/Downloads/hadoop-1.2.1-bin.tar.gz ./ (通过polysh从本地机器将安装包上传到虚拟机上,如果你没有把虚拟机的id_rsa.pub设置到本地机器的authorized_keys文件中,你需要输入本地机器登陆的密码-建议设置)
- ls hadoop*
hadoop@hadoop-slave01 : hadoop-1.2.1-bin.tar.gz
hadoop@hadoop-slave02 : hadoop-1.2.1-bin.tar.gz
hadoop@hadoop-master : hadoop-1.2.1-bin.tar.gz - 下载jdk安装包:jdk-8u111-linux-x64.tar.gz
- scp username@192.168.31.19:/Users/username/Downloads/jdk-8u111-linux-x64.tar.gz ./
3.2 hadoop安装
1.安装jdk和配置环境变量
tar -xzvf jdk-8u111-linux-x64.tar.gz
echo "export JAVA_HOME=/home/hadoop/workspace/jdk8" >> .bash_profile
echo "export PATH=$JAVA_HOME/bin:$PATH" >> .bash_profile
source .bash_profile
echo "export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar" >> .bash_profile
source .bash_profile
2.安装hadoop和修改配置
tar -xzvf hadoop-1.2.1-bin.tar.gz
echo "export HADOOP_HOME=/home/hadoop/workspace/hadoop" >> .bash_profile
source .bash_profile
echo "export PATH=$HADOOP_HOME/bin:$PATH" >> .bash_profile
source .bash_profile
//修改conf中配置文件
#修改hadoop-env.sh
#修改JAVA_HOME变量
# The java implementation to use. Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
export JAVA_HOME=/home/hadoop/workspace/jdk8
# Extra Java CLASSPATH elements. Optional.
# export HADOOP_CLASSPATH=
#修改core-site.xml
#设置namenode的地址和端口
#设置hadoop的中间结果和中间日志文件等的临时目录
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop-master:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/workspace/hadoop_dir/tmp</value>
</property>
</configuration>
#修改hdfs-site.xml
#配置namenode节点目录
#配置datanode节点目录
#配置备份数量
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/workspace/hadoop_dir/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/workspace/hadoop_dir/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
#修改mapred-site.xml
#配置map-reduce中的jobtracker的地址和端口
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hadoop-master:8021</value>
</property>
</configuration>
#修改masters
#配置master机器的hostname列表
hadoop-master
#修改slaves
#配置slave机器的hastname列表
hadoop-slave01
hadoop-slave02
三台虚拟机按照同样的方式配置成功后,hadoop就安装和配置成功了
hadoop服务启动
1.启动hadoop服务
- ssh hadoop@hadoop-master
- hadoop namenode -format (这个命令只需要执行一次)
正常格式化namenode输出结果如下:
************************************************************/
17/01/07 21:31:32 INFO util.GSet: Computing capacity for map BlocksMap
17/01/07 21:31:32 INFO util.GSet: VM type = 64-bit
17/01/07 21:31:32 INFO util.GSet: 2.0% max memory = 1013645312
17/01/07 21:31:32 INFO util.GSet: capacity = 2^21 = 2097152 entries
17/01/07 21:31:32 INFO util.GSet: recommended=2097152, actual=2097152
17/01/07 21:31:32 INFO namenode.FSNamesystem: fsOwner=hadoop
17/01/07 21:31:32 INFO namenode.FSNamesystem: supergroup=supergroup
17/01/07 21:31:32 INFO namenode.FSNamesystem: isPermissionEnabled=true
17/01/07 21:31:32 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
17/01/07 21:31:32 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
17/01/07 21:31:32 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
17/01/07 21:31:32 INFO namenode.NameNode: Caching file names occuring more than 10 times
17/01/07 21:31:33 INFO common.Storage: Image file /home/hadoop/workspace/hadoop_dir/name/current/fsimage of size 112 bytes saved in 0 seconds.
17/01/07 21:31:33 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/home/hadoop/workspace/hadoop_dir/name/current/edits
17/01/07 21:31:33 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/home/hadoop/workspace/hadoop_dir/name/current/edits
17/01/07 21:31:33 INFO common.Storage: Storage directory /home/hadoop/workspace/hadoop_dir/name has been successfully formatted.
17/01/07 21:31:33 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop-master/192.168.31.88
************************************************************/
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 设置hadoop-master到hadoop-slave01 hadoop-slave02的ssh免密登录
- cd workspace/hadoop/bin/
- sh start-all.sh
正常启动输入日志如下:
starting namenode, logging to /home/hadoop/workspace/hadoop/libexec/../logs/hadoop-hadoop-namenode-hadoop-master.out
hadoop-slave01: starting datanode, logging to /home/hadoop/workspace/hadoop/libexec/../logs/hadoop-hadoop-datanode-hadoop-slave01.out
hadoop-slave02: starting datanode, logging to /home/hadoop/workspace/hadoop/libexec/../logs/hadoop-hadoop-datanode-hadoop-slave02.out
hadoop-master: starting secondarynamenode, logging to /home/hadoop/workspace/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-hadoop-master.out
starting jobtracker, logging to /home/hadoop/workspace/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-hadoop-master.out
hadoop-slave02: starting tasktracker, logging to /home/hadoop/workspace/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-hadoop-slave02.out
hadoop-slave01: starting tasktracker, logging to /home/hadoop/workspace/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-hadoop-slave01.out
可以看出master会启动:namenode、secondarynamenode、jobtracker三个java进程
slave会启动:datanode、tasktracker两个java进程
- jps 查看是否和日志中启动的进程一致
hadoop@hadoop-master : 6549 SecondaryNameNode
hadoop@hadoop-master : 6629 JobTracker
hadoop@hadoop-master : 6358 NameNode
hadoop@hadoop-master : 6874 Jps
hadoop@hadoop-slave01 : 4096 DataNode
hadoop@hadoop-slave01 : 4197 TaskTracker
hadoop@hadoop-slave01 : 4253 Jps
hadoop@hadoop-slave02 : 5177 TaskTracker
hadoop@hadoop-slave02 : 5076 DataNode
hadoop@hadoop-slave02 : 5230 Jps
- 停止hadoop服务:sh stop-all.sh
stopping jobtracker
hadoop-slave02: stopping tasktracker
hadoop-slave01: stopping tasktracker
stopping namenode
hadoop-slave01: stopping datanode
hadoop-slave02: stopping datanode
hadoop-master: stopping secondarynamenode
2.状态查看网站
map-reduce任务情况:http://hadoop-master:50030/jobtracker.jsp
dfs使用情况:http://hadoop-master:50070/dfshealth.jsp
4 执行hadoop任务
百度实现一个wordcount的hadoop jar包任务。百度上很多不在此说明
1.上传本地文件到hdfs文件系统中
- hadoop fs -mkdir input (在hdfs文件系统上创建input文件夹)
- hadoop fs -put input/* input (将本地文件夹input中的文件上传到hdfs文件夹中)
- hadoop fs -ls input
hadoop@hadoop-master:~/workspace$ hadoop fs -ls input
Found 3 items
-rw-r--r-- 3 hadoop supergroup 2416 2017-01-07 22:58 /user/hadoop/input/wiki01
-rw-r--r-- 3 hadoop supergroup 3475 2017-01-07 22:58 /user/hadoop/input/wiki02
-rw-r--r-- 3 hadoop supergroup 4778 2017-01-07 22:58 /user/hadoop/input/wiki03
- scp wenyicao@192.168.31.19:/Users/wenyi/workspace/smart-industry-parent/hadoop-wordcount/target/hadoop-wordcount-1.0-SNAPSHOT.jar ./ (将本地的hadoop程序上传到虚拟机)
- hadoop jar hadoop-wordcount-1.0-SNAPSHOT.jar com.cweeyii.hadoop/WordCount input output (运行hadoop的wordcount单词计数程序)
hadoop调度和运行日志:
hadoop@hadoop-master:~/workspace$ hadoop jar hadoop-wordcount-1.0-SNAPSHOT.jar com.cweeyii.hadoop/WordCount input output
开始执行
input output
17/01/07 23:24:33 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
17/01/07 23:24:33 INFO util.NativeCodeLoader: Loaded the native-hadoop library
17/01/07 23:24:33 WARN snappy.LoadSnappy: Snappy native library not loaded
17/01/07 23:24:33 INFO mapred.FileInputFormat: Total input paths to process : 3
17/01/07 23:24:34 INFO mapred.JobClient: Running job: job_201701072322_0002
17/01/07 23:24:35 INFO mapred.JobClient: map 0% reduce 0%
17/01/07 23:24:41 INFO mapred.JobClient: map 33% reduce 0%
17/01/07 23:24:42 INFO mapred.JobClient: map 100% reduce 0%
17/01/07 23:24:49 INFO mapred.JobClient: map 100% reduce 33%
17/01/07 23:24:51 INFO mapred.JobClient: map 100% reduce 100%
17/01/07 23:24:51 INFO mapred.JobClient: Job complete: job_201701072322_0002
17/01/07 23:24:51 INFO mapred.JobClient: Counters: 30
17/01/07 23:24:51 INFO mapred.JobClient: Map-Reduce Framework
17/01/07 23:24:51 INFO mapred.JobClient: Spilled Records=3400
17/01/07 23:24:51 INFO mapred.JobClient: Map output materialized bytes=20878
17/01/07 23:24:51 INFO mapred.JobClient: Reduce input records=1700
17/01/07 23:24:51 INFO mapred.JobClient: Virtual memory (bytes) snapshot=7414968320
17/01/07 23:24:51 INFO mapred.JobClient: Map input records=31
17/01/07 23:24:51 INFO mapred.JobClient: SPLIT_RAW_BYTES=309
17/01/07 23:24:51 INFO mapred.JobClient: Map output bytes=17460
17/01/07 23:24:51 INFO mapred.JobClient: Reduce shuffle bytes=20878
17/01/07 23:24:51 INFO mapred.JobClient: Physical memory (bytes) snapshot=584658944
17/01/07 23:24:51 INFO mapred.JobClient: Map input bytes=10669
17/01/07 23:24:51 INFO mapred.JobClient: Reduce input groups=782
17/01/07 23:24:51 INFO mapred.JobClient: Combine output records=0
17/01/07 23:24:51 INFO mapred.JobClient: Reduce output records=782
17/01/07 23:24:51 INFO mapred.JobClient: Map output records=1700
17/01/07 23:24:51 INFO mapred.JobClient: Combine input records=0
17/01/07 23:24:51 INFO mapred.JobClient: CPU time spent (ms)=1950
17/01/07 23:24:51 INFO mapred.JobClient: Total committed heap usage (bytes)=498544640
17/01/07 23:24:51 INFO mapred.JobClient: File Input Format Counters
17/01/07 23:24:51 INFO mapred.JobClient: Bytes Read=10669
17/01/07 23:24:51 INFO mapred.JobClient: FileSystemCounters
17/01/07 23:24:51 INFO mapred.JobClient: HDFS_BYTES_READ=10978
17/01/07 23:24:51 INFO mapred.JobClient: FILE_BYTES_WRITTEN=285064
17/01/07 23:24:51 INFO mapred.JobClient: FILE_BYTES_READ=20866
17/01/07 23:24:51 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=7918
17/01/07 23:24:51 INFO mapred.JobClient: File Output Format Counters
17/01/07 23:24:51 INFO mapred.JobClient: Bytes Written=7918
17/01/07 23:24:51 INFO mapred.JobClient: Job Counters
17/01/07 23:24:51 INFO mapred.JobClient: Launched map tasks=3
17/01/07 23:24:51 INFO mapred.JobClient: Launched reduce tasks=1
17/01/07 23:24:51 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=9720
17/01/07 23:24:51 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
17/01/07 23:24:51 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=13500
17/01/07 23:24:51 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
17/01/07 23:24:51 INFO mapred.JobClient: Data-local map tasks=3
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- hadoop fs -ls output/ (查看执行结果)
# _SUCESS表示执行结果的状态
# _logs表示执行的日志
# part-00000表示执行结果
-rw-r--r-- 3 hadoop supergroup 0 2017-01-07 23:24 /user/hadoop/output/_SUCCESS
drwxr-xr-x - hadoop supergroup 0 2017-01-07 23:24 /user/hadoop/output/_logs
-rw-r--r-- 3 hadoop supergroup 7918 2017-01-07 23:24 /user/hadoop/output/part-00000
- hadoop fs -cat output/part-00000 (最终输出结果)
hadoop@hadoop-master:~/workspace$ hadoop fs -cat output/part-00000
"BeginnerQuestions." 1
"CamelCase" 1
"PopularMusic" 1
"Richard 1
"RichardWagner" 1
"TableOfContents" 1
"WiKi" 1
"Wiki 1
"Wiki" 1
"Wiki"). 1
"WikiNode" 1
"by 1
"corrected 1
"edit" 1
"editing 1
"fixed 1
"free 1
"history" 1
"link 1
"native" 1
"plain-vanilla" 1
"popular 1
"pretty" 1
"quick".[5][6][7] 1
"tag" 2
"the 2
"wiki 2