hadoop-yarn+jps+tmp+linux两个机制

最新推荐文章于 2022-08-27 15:56:51 发布

你拓哥

最新推荐文章于 2022-08-27 15:56:51 发布

阅读量247

点赞数

分类专栏： hadoop

本文链接：https://blog.csdn.net/nituoge/article/details/103338766

版权

hadoop 专栏收录该内容

11 篇文章 0 订阅

订阅专栏

一.搭建 yarn伪分布式

1、配置参数

[ruoze@rzdata001 ~]$ echo $HADOOP_HOME
/home/ruoze/app/hadoop
[ruoze@rzdata001 ~]$ cd app/hadoop
[ruoze@rzdata001 hadoop]$ cd etc/hadoop
[ruoze@rzdata001 hadoop]$ cp mapred-site.xml.template  mapred-site.xml
[ruoze@rzdata001 hadoop]$ vim mapred-site.xml   # 设置资源调度器，添加以下内容

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

[ruoze@rzdata001 hadoop]$
[ruoze@rzdata001 hadoop]$ vim yarn-site.xml    # 为防止云主机被挖矿或病毒，将yarn的默认端口8088端口改为38088

<configuration>
<!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>rzdata001:38088</value>
    </property>
</configuration>

[ruoze@rzdata001 hadoop]$
[ruoze@rzdata001 hadoop]$ jps
25107 Jps
9577 NameNode
9707 DataNode
9871 SecondaryNameNode
[ruoze@rzdata001 hadoop]$ stop-dfs.sh 
19/12/01 18:40:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping namenodes on [rzdata001]
rzdata001: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [rzdata001]
rzdata001: stopping secondarynamenode
19/12/01 18:40:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[ruoze@rzdata001 hadoop]$
[ruoze@rzdata001 hadoop]$ vim hadoop-env.sh   # 设置hadoop的tmp目录为自己用户下的tmp目录

export HADOOP_PID_DIR=/home/ruoze/tmp

[ruoze@rzdata001 hadoop]$ vim yarn-env.sh   # 设置yarn的tmp目录为自己用户下的tmp目录

export YARN_PID_DIR=/home/ruoze/tmp

2、启动

[ruoze@rzdata001 ~]$ start-dfs.sh  # 启动 hdfs
19/12/01 18:45:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [rzdata001]
rzdata001: starting namenode, logging to /home/ruoze/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-ruoze-namenode-rzdata001.out
localhost: starting datanode, logging to /home/ruoze/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-ruoze-datanode-rzdata001.out
Starting secondary namenodes [rzdata001]
rzdata001: starting secondarynamenode, logging to /home/ruoze/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-ruoze-secondarynamenode-rzdata001.out
19/12/01 18:45:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[ruoze@rzdata001 ~]$ jps
26352 DataNode
26515 SecondaryNameNode
26633 Jps
26222 NameNode
[ruoze@rzdata001 ~]$ start-yarn.sh   # 启动yarn
starting yarn daemons
starting resourcemanager, logging to /home/ruoze/app/hadoop-2.6.0-cdh5.16.2/logs/yarn-ruoze-resourcemanager-rzdata001.out
localhost: starting nodemanager, logging to /home/ruoze/app/hadoop-2.6.0-cdh5.16.2/logs/yarn-ruoze-nodemanager-rzdata001.out
[ruoze@rzdata001 ~]$ jps
26352 DataNode
26515 SecondaryNameNode
27093 Jps
26791 NodeManager       # yarn的从
26684 ResourceManager   # yarn的主
26222 NameNode
[ruoze@rzdata001 ~]$

二.跑mr count案例

1、创建样例数据

[ruoze@rzdata001 ~]$ cd data
[ruoze@rzdata001 data]$ vim worddata.txt

hello world! this is my house
jay jay music
music movie book
love and peace
hello world
rzdata
cong harry up
harry up
2 3 5 88
66 88 0

[ruoze@rzdata001 data]$

2、创建hdfs上input 目录，上传数据

[ruoze@rzdata001 data]$ 
[ruoze@rzdata001 data]$ hadoop fs -mkdir /user/ruoze/data/wordcount/input
19/12/01 18:52:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[ruoze@rzdata001 data]$ 
[ruoze@rzdata001 data]$ hadoop fs -put worddata.txt /user/ruoze/data/wordcount/input
19/12/01 18:53:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[ruoze@rzdata001 data]$

3、寻找wordcount 的jar包

[ruoze@rzdata001 data]$ cd 
[ruoze@rzdata001 ~]$ find ./ -name '*example*.jar'
./app/hadoop-2.6.0-cdh5.16.2/share/hadoop/mapreduce1/hadoop-examples-2.6.0-mr1-cdh5.16.2.jar
./app/hadoop-2.6.0-cdh5.16.2/share/hadoop/mapreduce2/sources/hadoop-mapreduce-examples-2.6.0-cdh5.16.2-test-sources.jar
./app/hadoop-2.6.0-cdh5.16.2/share/hadoop/mapreduce2/sources/hadoop-mapreduce-examples-2.6.0-cdh5.16.2-sources.jar
./app/hadoop-2.6.0-cdh5.16.2/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar   # 此处选择这个
[ruoze@rzdata001 ~]$

4、wordcount执行

[ruoze@rzdata001 ~]$ hadoop  jar \
> ./app/hadoop-2.6.0-cdh5.16.2/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar \
> wordcount \                               # jar包的主函数
> /user/ruoze/data/wordcount/input  \       # 输入目录，hdfs上已创建
> /user/ruoze/data/wordcount/output         # 输出目录，hdfs上未创建
19/12/01 19:04:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/12/01 19:04:08 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/12/01 19:04:09 INFO input.FileInputFormat: Total input paths to process : 1
19/12/01 19:04:09 INFO mapreduce.JobSubmitter: number of splits:1
19/12/01 19:04:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1575197144154_0001
19/12/01 19:04:10 INFO impl.YarnClientImpl: Submitted application application_1575197144154_0001
19/12/01 19:04:10 INFO mapreduce.Job: The url to track the job: http://rzdata001:38088/proxy/application_1575197144154_0001/
19/12/01 19:04:10 INFO mapreduce.Job: Running job: job_1575197144154_0001
19/12/01 19:04:16 INFO mapreduce.Job: Job job_1575197144154_0001 running in uber mode : false
19/12/01 19:04:16 INFO mapreduce.Job:  map 0% reduce 0%
19/12/01 19:04:20 INFO mapreduce.Job:  map 100% reduce 0%
19/12/01 19:04:26 INFO mapreduce.Job:  map 100% reduce 100%
19/12/01 19:04:27 INFO mapreduce.Job: Job job_1575197144154_0001 completed successfully
19/12/01 19:04:28 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=257
                FILE: Number of bytes written=286553
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=267
                HDFS: Number of bytes written=155
                HDFS: Number of read operations=6
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=1
                Launched reduce tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=2049
                Total time spent by all reduces in occupied slots (ms)=2808
                Total time spent by all map tasks (ms)=2049
                Total time spent by all reduce tasks (ms)=2808
                Total vcore-milliseconds taken by all map tasks=2049
                Total vcore-milliseconds taken by all reduce tasks=2808
                Total megabyte-milliseconds taken by all map tasks=2098176
                Total megabyte-milliseconds taken by all reduce tasks=2875392
        Map-Reduce Framework
                Map input records=10
                Map output records=30
                Map output bytes=255
                Map output materialized bytes=257
                Input split bytes=131
                Combine input records=30
                Combine output records=24
                Reduce input groups=24
                Reduce shuffle bytes=257
                Reduce input records=24
                Reduce output records=24
                Spilled Records=48
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=101
                CPU time spent (ms)=960
                Physical memory (bytes) snapshot=482902016
                Virtual memory (bytes) snapshot=5544710144
                Total committed heap usage (bytes)=479723520
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=136
        File Output Format Counters 
                Bytes Written=155
[ruoze@rzdata001 ~]$
[ruoze@rzdata001 ~]$ hadoop fs -ls /user/ruoze/data/wordcount/output
19/12/01 19:06:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   1 ruoze supergroup          0 2019-12-01 19:04 /user/ruoze/data/wordcount/output/_SUCCESS
-rw-r--r--   1 ruoze supergroup        155 2019-12-01 19:04 /user/ruoze/data/wordcount/output/part-r-00000
[ruoze@rzdata001 ~]$ 
[ruoze@rzdata001 ~]$ hadoop fs -cat /user/ruoze/data/wordcount/output/part-r-00000
19/12/01 19:06:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
0       1
2       1
3       1
5       1
66      1
88      2
and     1
book    1
cong    1
harry   2
hello   2
house   1
is      1
jay     2
love    1
movie   1
music   2
my      1
peace   1
rzdata  1
this    1
up      2
world   1
world!  1
[ruoze@rzdata001 ~]$

此时到可以到http://rzdata001:38088 查看对应job执行日志

5、再次理解 HDFS、MR、Yarn的区别：

HDFS：数据存储，计算结果又返回，存储在HDFS上
MR：计算框架，具体实现比如jar包，包含计算逻辑
Yarn：资源+作业调度

6、延伸

现在的大数据
存储：HDFS 分布式文件系统 Hive HBase Kudu Cassandra
计算：MR Hivesql Spark Flink
资源+作业调度： Yarn（1家）

三.整理 jps命令

jps是查看java进程的命令。用来查看当前java进程有哪些。不一定准哦~
最准的还是 ps -ef

3.1 所在位置

[ruoze@rzdata001 bin]$ which jps
/usr/java/jdk1.8.0_121/bin/jps
[ruoze@rzdata001 bin]$ jps
26352 DataNode
26515 SecondaryNameNode
26791 NodeManager
31771 Jps
26684 ResourceManager
26222 NameNode
[ruoze@rzdata001 bin]$ jps -l
26352 org.apache.hadoop.hdfs.server.datanode.DataNode
26515 org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
31798 sun.tools.jps.Jps
26791 org.apache.hadoop.yarn.server.nodemanager.NodeManager
26684 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
26222 org.apache.hadoop.hdfs.server.namenode.NameNode
[ruoze@rzdata001 bin]$

3.2 对应的进程的标识文件在哪？

各个用户不同目录：/tmp/hsperfdata_username

[root@rzdata001 tmp]# pwd
/tmp
[root@rzdata001 tmp]# ll hsperfdata_ruoze
total 160
-rw------- 1 ruoze ruoze 32768 Dec  1 19:25 26222
-rw------- 1 ruoze ruoze 32768 Dec  1 19:24 26352
-rw------- 1 ruoze ruoze 32768 Dec  1 19:24 26515
-rw------- 1 ruoze ruoze 32768 Dec  1 19:25 26684
-rw------- 1 ruoze ruoze 32768 Dec  1 19:24 26791
[root@rzdata001 tmp]#

3.3 作用

查询 pid 进程名称

3.4 process information unavailable CDH

进程所属的用户去执行 jps命令，只显示自己的相关的进程信息

[root@rzdata001 tmp]# su - ruoze  # 以进程启动的用户查看
Last login: Sun Dec  1 18:30:16 CST 2019 on pts/0
[ruoze@rzdata001 ~]$ jps
26352 DataNode
26515 SecondaryNameNode
435 Jps
26791 NodeManager
26684 ResourceManager
26222 NameNode
[ruoze@rzdata001 ~]$ 
[ruoze@rzdata001 ~]$ logout  # 以root用户查看
[root@rzdata001 tmp]# jps
26352 DataNode
26515 SecondaryNameNode
26791 NodeManager
26684 ResourceManager
508 Jps
26222 NameNode
[root@rzdata001 tmp]# 
[root@rzdata001 tmp]# su - cong   # 以其他用户查看
Last login: Sun Dec  1 19:26:47 CST 2019 on pts/1
[cong@rzdata001 ~]$ jps 
598 Jps
[cong@rzdata001 ~]$

** 坑 **
由于Linux的两个机制，会出现jps查看进程不准确的情况。

1、oom-kill机制和

若刚好本进程由于oom-killer被杀掉，jps查看显示有进程，但是实际已被kill。

2、/tmp默认存储周期1个月，过期会自动清空不在规则以内的文件

若刚好本进程文件被删除，jps查看没有进程，实际仍存在。若部署了进程自动重启脚本且脚本使用的jps，则会出现循环紊乱的情况

3.5 文件不影响进程的启动和停止

3.6 Linux机制 oom-killer机制

全称：Out of Memory (OOM) killer，它的作用是，当系统的内存用光的时候，系统内核会自动的Kill掉一个或者一些进程，以使系统能继续的恢复到正常的运行状态。

最近几年VPS和云主机开始流行，而VPS和云主机大部分都不会拥有太高的配置，再加上一些系统上软件的配置的不合理，可能会导致内存会出现用光的情况。

另外，有些时候编译程序也会需要足够的内存，比如编译PHP的时候小内存（印象中大概是在300MB左右临界，32位和64位情况会不同）会无法启用fileinfo模块，启用了内存就会爆掉。

OOM Killer每一次Kill掉进程都会在messages日志里留下记录，检查的命令为：
cat /var/log/messages | grep oom
部分系统时中日志里的关键字为：Out of memory，对应的检查命令为：
cat /var/log/messages | grep -i 'Out of memory'
如果日志里频繁的出现OOM记录，那么需要考虑调整服务器配置（软件或者硬件）。

3.7 Linux机制 /tmp存在默认存储周期，过期会自动清空不在规则以内的文件

[root@rzdata001 ~]# pwd
/root
[root@rzdata001 ~]# cd /usr/lib/tmpfiles.d/
[root@rzdata001 tmpfiles.d]# ll tmp.conf
-rw-r--r--  1 root root  638 Apr 11  2018 tmp.conf
[root@rzdata001 tmpfiles.d]# cat tmp.conf 
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# See tmpfiles.d(5) for details

# Clear tmp directories separately, to make them easier to override
v /tmp 1777 root root 10d     # 默认保留10天
v /var/tmp 1777 root root 30d

# Exclude namespace mountpoints created with PrivateTmp=yes
x /tmp/systemd-private-%b-*
X /tmp/systemd-private-%b-*/tmp
x /var/tmp/systemd-private-%b-*
X /var/tmp/systemd-private-%b-*/tmp
[root@rzdata001 tmpfiles.d]# 
[root@rzdata001 tmpfiles.d]# cd /tmp
[root@rzdata001 tmp]# ll 
total 0
drwxrwxr-x 2 rundeck rundeck  6 Nov 24 19:48 groovy-generated-4375842934200233345-tmpdir
drwxrwxr-x 2 rundeck rundeck  6 Nov 24 19:55 groovy-generated-5402246860382614329-tmpdir
drwxrwxr-x 2 rundeck rundeck  6 Nov 24 21:59 groovy-generated-566492547453384263-tmpdir
drwxrwxr-x 4 ruoze   ruoze   35 Dec  1 18:45 hadoop-ruoze
drwxr-xr-x 2 cong    cong     6 Dec  1 19:28 hsperfdata_cong
drwxr-xr-x 2 root    root     6 Dec  1 19:27 hsperfdata_root
drwxr-xr-x 2 rundeck rundeck  6 Nov 25 21:35 hsperfdata_rundeck
drwxr-xr-x 2 ruoze   ruoze   66 Dec  1 19:27 hsperfdata_ruoze
drwxrwxr-x 4 ruoze   ruoze   30 Dec  1 18:45 Jetty_0_0_0_0_50070_hdfs____w2cu08
drwxrwxr-x 4 ruoze   ruoze   30 Nov 28 21:46 Jetty_0_0_0_0_50090_secondary____y6aanv
drwxrwxr-x 5 ruoze   ruoze   43 Dec  1 18:45 Jetty_0_0_0_0_8042_node____19tj0x
drwxrwxr-x 4 ruoze   ruoze   30 Nov 28 22:05 Jetty_localhost_33426_datanode____jbjx8f
drwxrwxr-x 4 ruoze   ruoze   30 Nov 28 22:00 Jetty_localhost_35443_datanode____9p2ew
drwxrwxr-x 4 ruoze   ruoze   30 Nov 28 21:46 Jetty_localhost_45570_datanode____.xz2hnw
drwxrwxr-x 4 ruoze   ruoze   30 Dec  1 18:45 Jetty_localhost_49272_datanode____.a6py5t
drwxrwxr-x 5 ruoze   ruoze   43 Dec  1 18:45 Jetty_rzdata001_38088_cluster____zd862i
drwxrwxr-x 4 ruoze   ruoze   30 Dec  1 18:45 Jetty_rzdata001_50090_secondary____.22gr7l
---x-wx--- 1 root    root     0 Aug  8  2018 pcap_compile.lock
---------- 1 root    root     0 Aug  9  2018 signal.lock
drwx------ 3 root    root    16 Nov 28 21:14 systemd-private-e58030d12915446b9c6cc6e78655eafe-ntpd.service-tm4yof
[root@rzdata001 tmp]#

如果你不想让系统自动清理/tmp下以hsperfdata开头的目录，那么增加下面这条内容到配置文件中即可：