hadoop-2.6.0-cdh5.16.2部署

                         hadoop-2.6.0-cdh5.16.2 部署

 

1.首先先下载tar包并解压

[hadoop@xkhadoop software]$ wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.16.2/

[hadoop@xkhadoop software]$ ll
总用量 424176
-rw-r--r--. 1 hadoop hadoop 434354462 11月 29 00:10 hadoop-2.6.0-cdh5.16.2.tar.gz

[hadoop@xkhadoop software]$ tar -xzvf hadoop-2.6.0-cdh5.16.2.tar.gz -C ../app/

[hadoop@xkhadoop software]$ cd ../app/
[hadoop@xkhadoop app]$ ll
总用量 0
drwxr-xr-x. 14 hadoop hadoop 241 6月   3 19:11 hadoop-2.6.0-cdh5.16.2

2.部署hadoop-2.6.0-cdh5.16.2

部署环境：

        1.java最好1.8以上

[root@xkhadoop ~]# cd /usr/bin/

rz下jdk-8u45-linux-x64这个tar包

[root@xkhadoop bin]# tar -xzvf jdk-8u45-linux-x64.gz 

[root@xkhadoop bin]# ll|grep java
lrwxrwxrwx. 1 root root           22 11月 10 22:38 java -> /etc/alternatives/java
lrwxrwxrwx. 1 root root           24 11月 10 22:39 javaws -> /etc/alternatives/javaws
-rwxr-xr-x. 1 root root         5530 4月  11 2018 javaws.itweb
[root@xkhadoop bin]# rm -rf java

[root@xkhadoop bin]# chown root:root jdk1.8.0_45

[root@xkhadoop bin]# chown -R root:root jdk1.8.0_45/*

[root@xkhadoop bin]# ll |grep jdk
drwxr-xr-x. 8 root root          255 4月  11 2015 jdk1.8.0_45
-rw-r--r--. 1 root root    173271626 11月 28 23:02 jdk-8u45-linux-x64.gz
[root@xkhadoop bin]# cd jdk1.8.0_45/
[root@xkhadoop jdk1.8.0_45]# ll
总用量 25964
drwxr-xr-x. 2 root root     4096 4月  11 2015 bin
-r--r--r--. 1 root root     3244 4月  11 2015 COPYRIGHT
drwxr-xr-x. 4 root root      122 4月  11 2015 db
drwxr-xr-x. 3 root root      132 4月  11 2015 include
-rw-r--r--. 1 root root  5172312 3月  10 2015 javafx-src.zip
drwxr-xr-x. 5 root root      185 4月  11 2015 jre
drwxr-xr-x. 5 root root      245 4月  11 2015 lib
-r--r--r--. 1 root root       40 4月  11 2015 LICENSE
drwxr-xr-x. 4 root root       47 4月  11 2015 man
-r--r--r--. 1 root root      159 4月  11 2015 README.html
-rw-r--r--. 1 root root      525 4月  11 2015 release
-rw-r--r--. 1 root root 21099089 4月  11 2015 src.zip
-rw-r--r--. 1 root root   110114 3月  10 2015 THIRDPARTYLICENSEREADME-JAVAFX.txt
-r--r--r--. 1 root root   178403 4月  11 2015 THIRDPARTYLICENSEREADME.txt

 

[root@xkhadoop bin]# ln -s jdk1.8.0_45 java
[root@xkhadoop bin]# ll |grep java
lrwxrwxrwx. 1 root root           11 11月 29 23:04 java -> jdk1.8.0_45
lrwxrwxrwx. 1 root root           24 11月 10 22:39 javaws -> /etc/alternatives/javaws
-rwxr-xr-x. 1 root root         5530 4月  11 2018 javaws.itweb

[root@xkhadoop ~]# vim .bashrc 

export JAVA_HOME=/usr/bin/java
export PATH=$JAVA_HOME/bin:$PATH

[root@xkhadoop ~]# source .bashrc 

[root@xkhadoop ~]# which java
/usr/bin/java/bin/java
[root@xkhadoop ~]# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

 

        2.ssh信任关系

[hadoop@xkhadoop ~]$ ll -a
总用量 32
drwx------. 22 hadoop hadoop 4096 11月 29 22:09 .
drwxr-xr-x.  6 root   root     54 11月 24 15:52 ..
drwxrwxr-x.  3 hadoop hadoop   36 11月 29 22:11 app
-rw-------.  1 hadoop hadoop  187 11月 24 12:48 .bash_history
-rw-r--r--.  1 hadoop hadoop   18 8月   8 20:06 .bash_logout
-rw-r--r--.  1 hadoop hadoop  193 8月   8 20:06 .bash_profile
-rw-r--r--.  1 hadoop hadoop  231 8月   8 20:06 .bashrc
drwxrwxr-x. 14 hadoop hadoop 4096 11月 19 22:18 .cache
drwxrwxr-x. 14 hadoop hadoop  261 11月 19 22:18 .config
drwxrwxr-x.  2 hadoop hadoop    6 11月 18 21:18 data
drwx------.  3 hadoop hadoop   25 11月 19 22:17 .dbus
-rw-------.  1 hadoop hadoop   16 11月 19 22:17 .esd_auth
-rw-------.  1 hadoop hadoop  310 11月 19 22:17 .ICEauthority
drwxrwxr-x.  2 hadoop hadoop    6 11月 29 22:04 lib
drwx------.  3 hadoop hadoop   19 11月 19 22:17 .local
drwxrwxr-x.  2 hadoop hadoop    6 11月 29 22:04 log
drwxr-xr-x.  4 hadoop hadoop   39 11月 10 22:37 .mozilla
drwxrwxr-x.  2 hadoop hadoop   43 11月 29 22:09 software
drwxrwxr-x.  2 hadoop hadoop    6 11月 29 22:04 sourcecode
drwxrwxr-x.  2 hadoop hadoop    6 11月 29 22:04 tmp
drwxr-xr-x.  2 hadoop hadoop    6 11月 19 22:17 公共
drwxr-xr-x.  2 hadoop hadoop    6 11月 19 22:17 模板
drwxr-xr-x.  2 hadoop hadoop    6 11月 19 22:17 视频
drwxr-xr-x.  2 hadoop hadoop    6 11月 19 22:17 图片
drwxr-xr-x.  2 hadoop hadoop    6 11月 19 22:17 文档
drwxr-xr-x.  2 hadoop hadoop    6 11月 19 22:17 下载
drwxr-xr-x.  2 hadoop hadoop    6 11月 19 22:17 音乐
drwxr-xr-x.  2 hadoop hadoop    6 11月 19 22:17 桌面

没有ssh的隐藏文件则可以生成ssh文件，如果有.ssh文件，可以先mv,然后重新生成

[hadoop@xkhadoop ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:j8hC/3h0v/3h8tbUgW3QqZ7FxMPjSw1626B2ijEtF/I hadoop@xkhadoop
The key's randomart image is:
+---[RSA 2048]----+
|              + .|
|             ..O |
|             .O.+|
|          . oooO.|
|    .   S  +.+=++|
|   . o ..o= Eoo.+|
|    . +....O o o.|
|     . o. . oo..o|
|      ...   ..=+.|
+----[SHA256]-----+
[hadoop@xkhadoop ~]$ ll -a
总用量 32
drwx------. 23 hadoop hadoop 4096 11月 29 22:31 .
drwxr-xr-x.  6 root   root     54 11月 24 15:52 ..
drwxrwxr-x.  3 hadoop hadoop   36 11月 29 22:11 app
-rw-------.  1 hadoop hadoop  187 11月 24 12:48 .bash_history
-rw-r--r--.  1 hadoop hadoop   18 8月   8 20:06 .bash_logout
-rw-r--r--.  1 hadoop hadoop  193 8月   8 20:06 .bash_profile
-rw-r--r--.  1 hadoop hadoop  231 8月   8 20:06 .bashrc
drwxrwxr-x. 14 hadoop hadoop 4096 11月 19 22:18 .cache
drwxrwxr-x. 14 hadoop hadoop  261 11月 19 22:18 .config
drwxrwxr-x.  2 hadoop hadoop    6 11月 18 21:18 data
drwx------.  3 hadoop hadoop   25 11月 19 22:17 .dbus
-rw-------.  1 hadoop hadoop   16 11月 19 22:17 .esd_auth
-rw-------.  1 hadoop hadoop  310 11月 19 22:17 .ICEauthority
drwxrwxr-x.  2 hadoop hadoop    6 11月 29 22:04 lib
drwx------.  3 hadoop hadoop   19 11月 19 22:17 .local
drwxrwxr-x.  2 hadoop hadoop    6 11月 29 22:04 log
drwxr-xr-x.  4 hadoop hadoop   39 11月 10 22:37 .mozilla
drwxrwxr-x.  2 hadoop hadoop   43 11月 29 22:09 software
drwxrwxr-x.  2 hadoop hadoop    6 11月 29 22:04 sourcecode
drwx------.  2 hadoop hadoop   38 11月 29 22:31 .ssh
drwxrwxr-x.  2 hadoop hadoop    6 11月 29 22:04 tmp

 [hadoop@xkhadoop .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

修改权限：

[root@xkhadoop hadoop]# chmod 600 .ssh/authorized_keys 

[hadoop@xkhadoop .ssh]$ ssh xkhadoop date
The authenticity of host 'xkhadoop (192.168.182.128)' can't be established.
ECDSA key fingerprint is SHA256:Fhy0WViIcGq2zfCiRZ3DsYy6A+7L361pvzBE7nNcgyM.
ECDSA key fingerprint is MD5:c2:a1:61:72:6b:e6:07:fa:2d:e8:4e:13:24:94:ea:a2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'xkhadoop,192.168.182.128' (ECDSA) to the list of known hosts.
2019年 11月 29日 星期五 22:38:54 CST
[hadoop@xkhadoop .ssh]$ ssh xkhadoop date
2019年 11月 29日 星期五 22:38:59 CST

配置成功

 

创建软连接：

[hadoop@xkhadoop app]$ ln -s hadoop-2.6.0-cdh5.16.2 hadoop
[hadoop@xkhadoop app]$ ll
总用量 0
lrwxrwxrwx.  1 hadoop hadoop  22 11月 29 22:42 hadoop -> hadoop-2.6.0-cdh5.16.2
drwxr-xr-x. 14 hadoop hadoop 241 6月   3 19:11 hadoop-2.6.0-cdh5.16.2

配置环境变量：

[hadoop@xkhadoop ~]$ vim .bashrc 

export HADOOP_HOME=/home/hadoop/app/hadoop
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH

[hadoop@xkhadoop ~]$ source .bashrc 
[hadoop@xkhadoop ~]$ which hadoop
~/app/hadoop/bin/hadoop

修改Hadoop配置文件

[hadoop@xkhadoop hadoop]$ vim etc/hadoop/core-site.xml 

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://xkhadoop:9000</value>
    </property>
</configuration>

[hadoop@xkhadoop hadoop]$ vim etc/hadoop/hdfs-site.xml 

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

 

[hadoop@xkhadoop ~]$ vim app/hadoop/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/bin/java

格式化：

[hadoop@xkhadoop ~]$ which hdfs
~/app/hadoop/bin/hdfs

[hadoop@xkhadoop ~]$ hdfs namenode -format

/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at xkhadoop/192.168.182.128
************************************************************/

说明成功了

[hadoop@xkhadoop ~]$ which start-dfs.sh
~/app/hadoop/sbin/start-dfs.sh
[hadoop@xkhadoop ~]$ start-dfs.sh

[hadoop@xkhadoop ~]$ jps
5510 DataNode
6280 SecondaryNameNode
6059 NameNode
6395 Jps

3.寻找调优参数地址

http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.16.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

web：xkhadoop:50070

 

4.操作hdfs文件系统：

  在hdfs里面创建文件夹

[hadoop@xkhadoop data]$ hadoop fs -mkdir /data
19/11/29 23:29:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@xkhadoop data]$ hadoop fs -ls /
19/11/29 23:29:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2019-11-29 23:29 /data

上传文件到文件系统

[hadoop@xkhadoop ~]$ cd data/
[hadoop@xkhadoop data]$ echo "xk">xk.log
[hadoop@xkhadoop data]$ ll
总用量 4
-rw-rw-r--. 1 hadoop hadoop 3 11月 29 23:28 xk.log

 

[hadoop@xkhadoop data]$ hadoop fs -put xk.log /data/
19/11/29 23:33:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/11/29 23:33:23 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /data/xk.log._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1723)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3508)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:694)
        at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:219)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:507)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2272)

        at org.apache.hadoop.ipc.Client.call(Client.java:1504)
        at org.apache.hadoop.ipc.Client.call(Client.java:1441)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:231)
        at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:425)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
        at com.sun.proxy.$Proxy11.addBlock(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1875)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1671)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:790)
put: File /data/xk.log._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.

出现报错：

[hadoop@xkhadoop data]$ jps
5510 DataNode
6280 SecondaryNameNode
6059 NameNode
7023 Jps

发现有dn

然后在web上面发现没有dn,于是先关闭nn dn snn

[hadoop@xkhadoop data]$ stop-all.sh 

修改下配置文件：

[hadoop@xkhadoop ~]$ vim app/hadoop/etc/hadoop/slaves
xkhadoop

[hadoop@xkhadoop ~]$ vim app/hadoop/etc/hadoop/hdfs-site.xml 

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>xkhadoop:50090</value>
    </property>
 <property>
        <name>dfs.namenode.secondary.https-address</name>
        <value>xkhadoop:50091</value>
    </property>
</configuration>

重新格式化

[hadoop@xkhadoop ~]$ hdfs namenode -format

[hadoop@xkhadoop ~]$ start-dfs.sh
19/11/29 23:47:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [xkhadoop]
xkhadoop: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-hadoop-namenode-xkhadoop.out
xkhadoop: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-hadoop-datanode-xkhadoop.out
Starting secondary namenodes [xkhadoop]
xkhadoop: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-hadoop-secondarynamenode-xkhadoop.out
19/11/29 23:47:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@xkhadoop ~]$ jps
8096 DataNode
8368 Jps
8261 SecondaryNameNode
7964 NameNode

web页面发现dn有一台机器了

再尝试把文件存入hdfs

[hadoop@xkhadoop data]$ hadoop fs -mkdir /data
19/11/29 23:50:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@xkhadoop data]$ hadoop fs -put xk.log /data/
19/11/29 23:50:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@xkhadoop data]$ hadoop fs -ls /data/
19/11/29 23:51:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r--   1 hadoop supergroup          3 2019-11-29 23:50 /data/xk.log

从hdfs里面得到文件

[hadoop@xkhadoop data]$ ll
总用量 4
-rw-rw-r--. 1 hadoop hadoop 3 11月 29 23:28 xk.log
[hadoop@xkhadoop data]$ rm -rf xk.log 

[hadoop@xkhadoop data]$ hadoop fs -get /data/xk.log
19/11/29 23:51:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@xkhadoop data]$ ll
总用量 4
-rw-r--r--. 1 hadoop hadoop 3 11月 29 23:52 xk.log

5.启动yarn

修改配置文件

[hadoop@xkhadoop hadoop]$ cp mapred-site.xml.template mapred-site.xml

[hadoop@xkhadoop hadoop]$ vim mapred-site.xml

<configuration>
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

 

[hadoop@xkhadoop ~]$ start-yarn.sh 

[hadoop@xkhadoop ~]$ jps
3152 DataNode
3826 Jps
3475 ResourceManager
3318 SecondaryNameNode
3018 NameNode
3615 NodeManager

说明yarn已经启动

6.修改pid存放地址

先停掉所有的进程

[hadoop@xkhadoop hadoop]$ stop-all.sh

[hadoop@xkhadoop hadoop]$ vim hadoop-env.sh 

export HADOOP_PID_DIR=/home/hadoop/tmp

[hadoop@xkhadoop hadoop]$ vim yarn-env.sh 

export YARN_PID_DIR=/home/hadoop/tmp

然后重启所有hadoop进程

[hadoop@xkhadoop hadoop]$ start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
19/11/30 22:48:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [xkhadoop]
xkhadoop: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-hadoop-namenode-xkhadoop.out
xkhadoop: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-hadoop-datanode-xkhadoop.out
Starting secondary namenodes [xkhadoop]
xkhadoop: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-hadoop-secondarynamenode-xkhadoop.out
19/11/30 22:48:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.16.2/logs/yarn-hadoop-resourcemanager-xkhadoop.out
xkhadoop: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.16.2/logs/yarn-hadoop-nodemanager-xkhadoop.out

[hadoop@xkhadoop hadoop]$ jps
4593 NameNode
6789 ResourceManager
6918 NodeManager
4727 DataNode
7225 Jps
4892 SecondaryNameNode

[hadoop@xkhadoop tmp]$ ll
总用量 20
-rw-rw-r--. 1 hadoop hadoop 5 11月 30 22:48 hadoop-hadoop-datanode.pid
-rw-rw-r--. 1 hadoop hadoop 5 11月 30 22:48 hadoop-hadoop-namenode.pid
-rw-rw-r--. 1 hadoop hadoop 5 11月 30 22:48 hadoop-hadoop-secondarynamenode.pid
-rw-rw-r--. 1 hadoop hadoop 5 11月 30 22:48 yarn-hadoop-nodemanager.pid
-rw-rw-r--. 1 hadoop hadoop 5 11月 30 22:48 yarn-hadoop-resourcemanager.pid

则说明成功了

yarn的地址：http://xkhadoop:8088/

7.跑一个WC MRjob

寻找Hadoop提供mr的jar包

[hadoop@xkhadoop hadoop]$ find ./ -name '*example*'.jar
./share/hadoop/mapreduce1/hadoop-examples-2.6.0-mr1-cdh5.16.2.jar
./share/hadoop/mapreduce2/sources/hadoop-mapreduce-examples-2.6.0-cdh5.16.2-test-sources.jar
./share/hadoop/mapreduce2/sources/hadoop-mapreduce-examples-2.6.0-cdh5.16.2-sources.jar
./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar

写一些数据然后把数据放进hdfs

[hadoop@xkhadoop data]$ vim 1.log

1
3
43
456
86 34 33 11 1 3 1 3 1 3

[hadoop@xkhadoop data]$ hadoop fs -mkdir /input
19/11/30 22:56:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@xkhadoop data]$ hadoop fs -mkdir /output
19/11/30 22:56:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@xkhadoop data]$ hadoop fs -ls /
19/11/30 22:56:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 3 items
drwxr-xr-x   - hadoop supergroup          0 2019-11-29 23:50 /data
drwxr-xr-x   - hadoop supergroup          0 2019-11-30 22:56 /input
drwxr-xr-x   - hadoop supergroup          0 2019-11-30 22:56 /output

[hadoop@xkhadoop data]$ hadoop fs -put 1.log /input/
19/11/30 22:56:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@xkhadoop data]$ hadoop fs -ls /input
19/11/30 22:57:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r--   1 hadoop supergroup         35 2019-11-30 22:56 /input/1.log

 

[hadoop@xkhadoop data]$ hadoop jar /home/hadoop/app/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar  wordcount /input /output1

[hadoop@xkhadoop data]$ hadoop fs -ls /output1
19/11/30 23:06:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2019-11-30 23:05 /output1/_SUCCESS
-rw-r--r--   1 hadoop supergroup         39 2019-11-30 23:05 /output1/part-r-00000

[hadoop@xkhadoop data]$ hadoop fs -cat /output1/part-r-00000
19/11/30 23:07:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
1       4
11      1
3       4
33      1
34      1
43      1
456     1
86      1