yarn部署及Linux的OOM、clean机制

YARN伪分布式

Yarn伪分布式部署 单点 主从架构
ResourceManager
NodeManager

Configure parameters as follows:参数配置如下
[pxj@pxj31 /home/pxj/app/hadoop/etc/hadoop]$vim mapred-site.xml
[pxj@pxj31 /home/pxj/app/hadoop/etc/hadoop]$cp mapred-site.xml.template  mapred-site.xml
[pxj@pxj31 /home/pxj/app/hadoop/etc/hadoop]$vim mapred-site.xml
<configuration>
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>
[pxj@pxj31 /home/pxj/app/hadoop/etc/hadoop]$ vim yarn-site.xml
<configuration>
<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>pxj31:38088</value>
    </property>
</configuration>
注意:
云主机 挖矿 病毒
yarn 8088端口
web界面防火墙 38088 50070 开放这两个端口

启动hdfs和yarn

[pxj@pxj31 /home/pxj]$start-dfs.sh 
19/12/01 12:50:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [pxj31]
pxj31: starting namenode, logging to /home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-pxj-namenode-pxj31.out
pxj31: starting datanode, logging to /home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-pxj-datanode-pxj31.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-pxj-secondarynamenode-pxj31.out
19/12/01 12:50:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[pxj@pxj31 /home/pxj]$jps
3842 DataNode
3733 NameNode
4041 SecondaryNameNode
4158 Jps
[pxj@pxj31 /home/pxj]$start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs/yarn-pxj-resourcemanager-pxj31.out
pxj31: starting nodemanager, logging to /home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs/yarn-pxj-nodemanager-pxj31.out

wordcount 单词统计

[pxj@pxj31 /home/pxj/app/hadoop]$find ./ -name '*example*.jar'
./share/hadoop/mapreduce1/hadoop-examples-2.6.0-mr1-cdh5.16.2.jar
./share/hadoop/mapreduce2/sources/hadoop-mapreduce-examples-2.6.0-cdh5.16.2-test-sources.jar
./share/hadoop/mapreduce2/sources/hadoop-mapreduce-examples-2.6.0-cdh5.16.2-sources.jar
./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar
查看所需的参数:
[pxj@pxj31 /home/pxj/app/hadoop]$hadoop jar ./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar wordcount
Usage: wordcount <in> [<in>...] <out>
编辑文件
[pxj@pxj31 /home/pxj/data]$cat 1.log 
pxj
wfy
cdr
lzh
zcl
zmj
lx
bjs
crh
dd
yy
kk
llll
uu
pp
jjj
上传文件
[pxj@pxj31 /home/pxj/data]$hadoop fs -put 1.log /a
19/12/01 13:01:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[pxj@pxj31 /home/pxj/data]$hadoop fs -ls /a
19/12/01 13:01:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r--   1 pxj supergroup         60 2019-12-01 13:01 /a/1.log
[pxj@pxj31 /home/pxj/app/hadoop]$hadoop jar \
> ./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar \
> wordcount \
> /a  /a/output
19/12/01 13:07:28 INFO mapreduce.Job: Counters: 49
    File Input Format Counters 
        Bytes Read=60
    File Output Format Counters 
        Bytes Written=91
[pxj@pxj31 /home/pxj/app/hadoop]$hadoop fs -cat /a/output/part-r-00000
19/12/01 13:14:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
bjs 1
cdr 1
crh 1
dd  1
jjj 1
kk  1
llll    1
lx  1
lzh 1
pp  1
pxj 1
uu  1
wfy 1
yy  1
zcl 1
zmj 1

总结

HDFS 存储 计算结果又返回存储HDFS
MR jar包 计算逻辑
Yarn 资源+作业调度
现在的大数据
存储 :HDFS 分布式文件系统 Hive HBase Kudu Cassandra
计算 :MR Hivesql Spark Flink
资源+作业调度 : Yarn(1家)

修改机器名称

[pxj@pxj31 /home/pxj/app/hadoop]$hostnamectl
   Static hostname: pxj31
         Icon name: computer-vm
           Chassis: vm
        Machine ID: b6683ab4fdc8441dab465b98a84b5d85
           Boot ID: 795382f68dde4216a68fbc087d047afd
    Virtualization: vmware
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-693.el7.x86_64
      Architecture: x86-64
[root@pxj31 /root]#hostnamectl set-hostname wfy
[root@pxj31 /root]#hostname
wfy
[pxj@pxj31 /home/pxj]$vi /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.25.31 pxj31

jps真正含义(重点)

jps位置

[pxj@pxj31 /home/pxj]$which jps
/usr/java/jdk1.8.0_121/bin/jps
[pxj@pxj31 /home/pxj]$jps
8049 NodeManager
7796 SecondaryNameNode
7624 DataNode
8440 Jps
7946 ResourceManager
7486 NameNode
[pxj@pxj31 /home/pxj]$jps -help
usage: jps [-help]
       jps [-q] [-mlvV] [<hostid>]
Definitions:
    <hostid>:      <hostname>[:<port>]
[pxj@pxj31 /home/pxj]$jps -l
8049 org.apache.hadoop.yarn.server.nodemanager.NodeManager
7796 org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
8468 sun.tools.jps.Jps
7624 org.apache.hadoop.hdfs.server.datanode.DataNode
7946 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
7486 org.apache.hadoop.hdfs.server.namenode.NameNode
[pxj@pxj31 /home/pxj]$jps -m
8049 NodeManager
8483 Jps -m
7796 SecondaryNameNode
7624 DataNode
7946 ResourceManager
7486 NameNode

对应的进程的标识文件在哪?

[pxj@pxj31 /tmp]$ll
drwxr-xr-x. 2 pxj pxj 66 12月 1 13:38 hsperfdata_pxj

总结

进程是pxj启动的 /tmp/hsperfdata_username

作用

查询 pid 进程名称
[pxj@pxj31 /tmp]$jps -l
8049 org.apache.hadoop.yarn.server.nodemanager.NodeManager
7796 org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
7624 org.apache.hadoop.hdfs.server.datanode.DataNode
8920 sun.tools.jps.Jps
7946 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
7486 org.apache.hadoop.hdfs.server.namenode.NameNode
如上代码:8049为pid
 process information unavailable CDH 

进程所属的用户 去执行 jps命令,只显示自己的相关的进程信息
root用户 看所有的,但是显示不可用

真假判断 :

[pxj@pxj31 /tmp/hsperfdata_pxj]$ps -ef|grep 7486
pxj 7486 1 0 13:34 ? 00:00:08 /usr/java/jdk1.8.0_121/bin/java -Dproc_namenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2 -Dhadoop.id.str=pxj -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs -Dhadoop.log.file=hadoop-pxj-namenode-pxj31.log -Dhadoop.home.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2 -Dhadoop.id.str=pxj -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode
pxj 9152 7153 0 14:01 pts/0 00:00:00 grep --color=auto 7486

通过统计确定是否存在

[pxj@pxj31 /tmp/hsperfdata_pxj]$ps -ef|grep 8049|grep -v grep|wc -l
1

文件 不影响 进程的启动和停止

证明:
1.删除/tmp下的文件
[pxj@pxj31 /tmp/hsperfdata_pxj]$jps
11312 Jps
8049 NodeManager
7796 SecondaryNameNode
7624 DataNode
7946 ResourceManager
7486 NameNode
[pxj@pxj31 /tmp/hsperfdata_pxj]$ll
总用量 160
-rw-------. 1 pxj pxj 32768 12月 1 14:33 7486
-rw-------. 1 pxj pxj 32768 12月 1 14:33 7624
-rw-------. 1 pxj pxj 32768 12月 1 14:33 7796
-rw-------. 1 pxj pxj 32768 12月 1 14:33 7946
-rw-------. 1 pxj pxj 32768 12月 1 14:33 8049
[pxj@pxj31 /tmp/hsperfdata_pxj]$rm -fr 7846
2.查看结果
[pxj@pxj31 /home/pxj]$jps
8049 NodeManager
7796 SecondaryNameNode
7624 DataNode
11464 Jps
7946 ResourceManager
[pxj@pxj31 /home/pxj]$ps -ef|grep -i  NameNode
pxj        7486      1  0 13:34 ?        00:00:15 /usr/java/jdk1.8.0_121/bin/java -Dproc_namenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2 -Dhadoop.id.str=pxj -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs -Dhadoop.log.file=hadoop-pxjnamenode-pxj31.log -Dhadoop.home.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2 -Dhadoop.id.str=pxj -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode
pxj        7796      1  0 13:35 ?        00:00:07 /usr/java/jdk1.8.0_121/bin/java -Dproc_secondarynamenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2 -Dhadoop.id.str=pxj -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs -Dhadoop.log.file=hadoop-pxj-secondarynamenode-pxj31.log -Dhadoop.home.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2 -Dhadoop.id.str=pxj -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
pxj       11491  11386  0 14:36 pts/1    00:00:00 grep --color=auto -i NameNode
3.停止
[pxj@pxj31 /home/pxj]$stop-all.sh 
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
19/12/01 14:37:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping namenodes on [pxj31]
pxj31: stopping namenode
pxj31: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
19/12/01 14:37:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
stopping yarn daemons
stopping resourcemanager
pxj31: stopping nodemanager
pxj31: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
no proxyserver to stop

结论:不影响进程启动和停止,但影响JPS判断进程,建议在生产使用ps -ef|grep 进程

Linux机制 oom-kill机制

某个进程 memory 使用过高,机器为了保护自己,放置夯住,去杀死内存使用最多的进程。
free -m 查看机器的内存情况
1.模拟杀死进程
[pxj@pxj31 /tmp/hsperfdata_pxj]$kill -9 12693
2.手动维护
[pxj@pxj31 /tmp/hsperfdata_pxj]$start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs/yarn-pxj-resourcemanager-pxj31.out
pxj31: nodemanager running as process 12797. Stop it first.
3.总结:
工作中如果某个进程挂了,重新启动后一段时间还是挂,要意识到:
以后进程挂了--》log位置--》error: 有error具体分析;没error机制就要 想到oom机制
free -m
cat /var/log/messages | grep oom
kill log info 断了 cat log找error
VM 2G CDH mysql+server ok 挂了 
意识: 以后进程挂了--》log位置--》error: 有error具体分析;没 想到oom机制
free -m
cat /var/log/messages | grep oom

Linux机制 /tmp默认存储周期 1个月 会自动清空不在规则以内的文件(重点)

[pxj@pxj31 /tmp]$cat yarn-pxj-resourcemanager.pid 
12693
[pxj@pxj31 /tmp]$cat yarn-pxj-resourcemanager.pid 
13472
修改hdfs的PID路径

[pxj@pxj31 /home/pxj/app/hadoop/etc/hadoop]$vim hadoop-env.sh
export HADOOP_PID_DIR=/home/pxj/tmp
修改yarn的PID路径

[pxj@pxj31 /home/pxj/app/hadoop/etc/hadoop]$vim yarn-env.sh
export YARN_PID_DIR=/home/pxj/tmp

参考网站:
https://www.yundaiwei.com/post/508.html

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值