YARN伪分布式
Yarn伪分布式部署 单点 主从架构
ResourceManager
NodeManager
Configure parameters as follows:参数配置如下
[pxj@pxj31 /home/pxj/app/hadoop/etc/hadoop]$vim mapred-site.xml
[pxj@pxj31 /home/pxj/app/hadoop/etc/hadoop]$cp mapred-site.xml.template mapred-site.xml
[pxj@pxj31 /home/pxj/app/hadoop/etc/hadoop]$vim mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
[pxj@pxj31 /home/pxj/app/hadoop/etc/hadoop]$ vim yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>pxj31:38088</value>
</property>
</configuration>
注意:
云主机 挖矿 病毒
yarn 8088端口
web界面防火墙 38088 50070 开放这两个端口
启动hdfs和yarn
[pxj@pxj31 /home/pxj]$start-dfs.sh
19/12/01 12:50:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [pxj31]
pxj31: starting namenode, logging to /home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-pxj-namenode-pxj31.out
pxj31: starting datanode, logging to /home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-pxj-datanode-pxj31.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-pxj-secondarynamenode-pxj31.out
19/12/01 12:50:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[pxj@pxj31 /home/pxj]$jps
3842 DataNode
3733 NameNode
4041 SecondaryNameNode
4158 Jps
[pxj@pxj31 /home/pxj]$start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs/yarn-pxj-resourcemanager-pxj31.out
pxj31: starting nodemanager, logging to /home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs/yarn-pxj-nodemanager-pxj31.out
wordcount 单词统计
[pxj@pxj31 /home/pxj/app/hadoop]$find ./ -name '*example*.jar'
./share/hadoop/mapreduce1/hadoop-examples-2.6.0-mr1-cdh5.16.2.jar
./share/hadoop/mapreduce2/sources/hadoop-mapreduce-examples-2.6.0-cdh5.16.2-test-sources.jar
./share/hadoop/mapreduce2/sources/hadoop-mapreduce-examples-2.6.0-cdh5.16.2-sources.jar
./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar
查看所需的参数:
[pxj@pxj31 /home/pxj/app/hadoop]$hadoop jar ./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar wordcount
Usage: wordcount <in> [<in>...] <out>
编辑文件
[pxj@pxj31 /home/pxj/data]$cat 1.log
pxj
wfy
cdr
lzh
zcl
zmj
lx
bjs
crh
dd
yy
kk
llll
uu
pp
jjj
上传文件
[pxj@pxj31 /home/pxj/data]$hadoop fs -put 1.log /a
19/12/01 13:01:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[pxj@pxj31 /home/pxj/data]$hadoop fs -ls /a
19/12/01 13:01:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r-- 1 pxj supergroup 60 2019-12-01 13:01 /a/1.log
[pxj@pxj31 /home/pxj/app/hadoop]$hadoop jar \
> ./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar \
> wordcount \
> /a /a/output
19/12/01 13:07:28 INFO mapreduce.Job: Counters: 49
File Input Format Counters
Bytes Read=60
File Output Format Counters
Bytes Written=91
[pxj@pxj31 /home/pxj/app/hadoop]$hadoop fs -cat /a/output/part-r-00000
19/12/01 13:14:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
bjs 1
cdr 1
crh 1
dd 1
jjj 1
kk 1
llll 1
lx 1
lzh 1
pp 1
pxj 1
uu 1
wfy 1
yy 1
zcl 1
zmj 1
总结
HDFS 存储 计算结果又返回存储HDFS
MR jar包 计算逻辑
Yarn 资源+作业调度
现在的大数据
存储 :HDFS 分布式文件系统 Hive HBase Kudu Cassandra
计算 :MR Hivesql Spark Flink
资源+作业调度 : Yarn(1家)
修改机器名称
[pxj@pxj31 /home/pxj/app/hadoop]$hostnamectl
Static hostname: pxj31
Icon name: computer-vm
Chassis: vm
Machine ID: b6683ab4fdc8441dab465b98a84b5d85
Boot ID: 795382f68dde4216a68fbc087d047afd
Virtualization: vmware
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-693.el7.x86_64
Architecture: x86-64
[root@pxj31 /root]#hostnamectl set-hostname wfy
[root@pxj31 /root]#hostname
wfy
[pxj@pxj31 /home/pxj]$vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.25.31 pxj31
jps真正含义(重点)
jps位置
[pxj@pxj31 /home/pxj]$which jps
/usr/java/jdk1.8.0_121/bin/jps
[pxj@pxj31 /home/pxj]$jps
8049 NodeManager
7796 SecondaryNameNode
7624 DataNode
8440 Jps
7946 ResourceManager
7486 NameNode
[pxj@pxj31 /home/pxj]$jps -help
usage: jps [-help]
jps [-q] [-mlvV] [<hostid>]
Definitions:
<hostid>: <hostname>[:<port>]
[pxj@pxj31 /home/pxj]$jps -l
8049 org.apache.hadoop.yarn.server.nodemanager.NodeManager
7796 org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
8468 sun.tools.jps.Jps
7624 org.apache.hadoop.hdfs.server.datanode.DataNode
7946 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
7486 org.apache.hadoop.hdfs.server.namenode.NameNode
[pxj@pxj31 /home/pxj]$jps -m
8049 NodeManager
8483 Jps -m
7796 SecondaryNameNode
7624 DataNode
7946 ResourceManager
7486 NameNode
对应的进程的标识文件在哪?
[pxj@pxj31 /tmp]$ll
drwxr-xr-x. 2 pxj pxj 66 12月 1 13:38 hsperfdata_pxj
总结
进程是pxj启动的 /tmp/hsperfdata_username
作用
查询 pid 进程名称
[pxj@pxj31 /tmp]$jps -l
8049 org.apache.hadoop.yarn.server.nodemanager.NodeManager
7796 org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
7624 org.apache.hadoop.hdfs.server.datanode.DataNode
8920 sun.tools.jps.Jps
7946 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
7486 org.apache.hadoop.hdfs.server.namenode.NameNode
如上代码:8049为pid
process information unavailable CDH
进程所属的用户 去执行 jps命令,只显示自己的相关的进程信息
root用户 看所有的,但是显示不可用
真假判断 :
[pxj@pxj31 /tmp/hsperfdata_pxj]$ps -ef|grep 7486
pxj 7486 1 0 13:34 ? 00:00:08 /usr/java/jdk1.8.0_121/bin/java -Dproc_namenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2 -Dhadoop.id.str=pxj -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs -Dhadoop.log.file=hadoop-pxj-namenode-pxj31.log -Dhadoop.home.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2 -Dhadoop.id.str=pxj -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode
pxj 9152 7153 0 14:01 pts/0 00:00:00 grep --color=auto 7486
通过统计确定是否存在
[pxj@pxj31 /tmp/hsperfdata_pxj]$ps -ef|grep 8049|grep -v grep|wc -l
1
文件 不影响 进程的启动和停止
证明:
1.删除/tmp下的文件
[pxj@pxj31 /tmp/hsperfdata_pxj]$jps
11312 Jps
8049 NodeManager
7796 SecondaryNameNode
7624 DataNode
7946 ResourceManager
7486 NameNode
[pxj@pxj31 /tmp/hsperfdata_pxj]$ll
总用量 160
-rw-------. 1 pxj pxj 32768 12月 1 14:33 7486
-rw-------. 1 pxj pxj 32768 12月 1 14:33 7624
-rw-------. 1 pxj pxj 32768 12月 1 14:33 7796
-rw-------. 1 pxj pxj 32768 12月 1 14:33 7946
-rw-------. 1 pxj pxj 32768 12月 1 14:33 8049
[pxj@pxj31 /tmp/hsperfdata_pxj]$rm -fr 7846
2.查看结果
[pxj@pxj31 /home/pxj]$jps
8049 NodeManager
7796 SecondaryNameNode
7624 DataNode
11464 Jps
7946 ResourceManager
[pxj@pxj31 /home/pxj]$ps -ef|grep -i NameNode
pxj 7486 1 0 13:34 ? 00:00:15 /usr/java/jdk1.8.0_121/bin/java -Dproc_namenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2 -Dhadoop.id.str=pxj -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs -Dhadoop.log.file=hadoop-pxjnamenode-pxj31.log -Dhadoop.home.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2 -Dhadoop.id.str=pxj -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode
pxj 7796 1 0 13:35 ? 00:00:07 /usr/java/jdk1.8.0_121/bin/java -Dproc_secondarynamenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2 -Dhadoop.id.str=pxj -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs -Dhadoop.log.file=hadoop-pxj-secondarynamenode-pxj31.log -Dhadoop.home.dir=/home/pxj/app/hadoop-2.6.0-cdh5.16.2 -Dhadoop.id.str=pxj -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/home/pxj/app/hadoop-2.6.0-cdh5.16.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
pxj 11491 11386 0 14:36 pts/1 00:00:00 grep --color=auto -i NameNode
3.停止
[pxj@pxj31 /home/pxj]$stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
19/12/01 14:37:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping namenodes on [pxj31]
pxj31: stopping namenode
pxj31: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
19/12/01 14:37:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
stopping yarn daemons
stopping resourcemanager
pxj31: stopping nodemanager
pxj31: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
no proxyserver to stop
结论:不影响进程启动和停止,但影响JPS判断进程,建议在生产使用ps -ef|grep 进程
Linux机制 oom-kill机制
某个进程 memory 使用过高,机器为了保护自己,放置夯住,去杀死内存使用最多的进程。
free -m 查看机器的内存情况
1.模拟杀死进程
[pxj@pxj31 /tmp/hsperfdata_pxj]$kill -9 12693
2.手动维护
[pxj@pxj31 /tmp/hsperfdata_pxj]$start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/pxj/app/hadoop-2.6.0-cdh5.16.2/logs/yarn-pxj-resourcemanager-pxj31.out
pxj31: nodemanager running as process 12797. Stop it first.
3.总结:
工作中如果某个进程挂了,重新启动后一段时间还是挂,要意识到:
以后进程挂了--》log位置--》error: 有error具体分析;没error机制就要 想到oom机制
free -m
cat /var/log/messages | grep oom
kill log info 断了 cat log找error
VM 2G CDH mysql+server ok 挂了
意识: 以后进程挂了--》log位置--》error: 有error具体分析;没 想到oom机制
free -m
cat /var/log/messages | grep oom
Linux机制 /tmp默认存储周期 1个月 会自动清空不在规则以内的文件(重点)
[pxj@pxj31 /tmp]$cat yarn-pxj-resourcemanager.pid
12693
[pxj@pxj31 /tmp]$cat yarn-pxj-resourcemanager.pid
13472
修改hdfs的PID路径
[pxj@pxj31 /home/pxj/app/hadoop/etc/hadoop]$vim hadoop-env.sh
export HADOOP_PID_DIR=/home/pxj/tmp
修改yarn的PID路径
[pxj@pxj31 /home/pxj/app/hadoop/etc/hadoop]$vim yarn-env.sh
export YARN_PID_DIR=/home/pxj/tmp
参考网站:
https://www.yundaiwei.com/post/508.html