8.hadoop3 启动的流程以及现象

已于 2024-08-22 15:56:23 修改
阅读量421 收藏 3
点赞数 3
分类专栏： hadoop 文章标签： eclipse java ide
于 2024-08-22 15:28:05 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/hxsln11/article/details/141429266
版权

1.102上    sbin/start-dfs.sh

[atguigu@hadoop102 ~]$ cd /opt/module/hadoop-3.1.3
[atguigu@hadoop102 hadoop-3.1.3]$
[atguigu@hadoop102 hadoop-3.1.3]$
[atguigu@hadoop102 hadoop-3.1.3]$ sbin/start-dfs.sh
Starting namenodes on [hadoop102]
hadoop102: namenode is running as process 2013.  Stop it first.
Starting datanodes
hadoop102: datanode is running as process 2086.  Stop it first.
hadoop104: WARNING: /opt/module/hadoop-3.1.3/logs does not exist. Creating.
hadoop103: datanode is running as process 2020.  Stop it first.
hadoop104: datanode is running as process 2162.  Stop it first.
Starting secondary namenodes [hadoop104]

103上  sbin/start-yarn.sh

[atguigu@hadoop103 ~]$ cd /opt/module/hadoop-3.1.3
[atguigu@hadoop103 hadoop-3.1.3]$
[atguigu@hadoop103 hadoop-3.1.3]$
[atguigu@hadoop103 hadoop-3.1.3]$
[atguigu@hadoop103 hadoop-3.1.3]$ sbin/start-yarn.sh

启动后jps查看

102

103

104

hadoop网址

http://hadoop102:9870/explorer.html#/

http://hadoop103:8088/cluster/nodes

检验mapreduce是否正常工作

[atguigu@hadoop102 hadoop-3.1.3]$ pwd
/opt/module/hadoop-3.1.3
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount  /wcinput  /wcoutput

失败的日志

[atguigu@hadoop102 hadoop-3.1.3]$ pwd
/opt/module/hadoop-3.1.3
[atguigu@hadoop102 hadoop-3.1.3]$
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount  /wcinput  /wcoutput
2024-08-22 15:47:02,220 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.188.103:8032
2024-08-22 15:47:04,195 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/atguigu/.staging/job_1724310448291_0001
2024-08-22 15:47:04,613 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 15:47:06,077 INFO input.FileInputFormat: Total input files to process : 1
2024-08-22 15:47:06,141 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 15:47:06,214 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 15:47:06,262 INFO mapreduce.JobSubmitter: number of splits:1
2024-08-22 15:47:06,682 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 15:47:06,867 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1724310448291_0001
2024-08-22 15:47:06,867 INFO mapreduce.JobSubmitter: Executing with tokens: []
2024-08-22 15:47:07,792 INFO conf.Configuration: resource-types.xml not found
2024-08-22 15:47:07,792 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2024-08-22 15:47:08,302 INFO impl.YarnClientImpl: Submitted application application_1724310448291_0001
2024-08-22 15:47:08,619 INFO mapreduce.Job: The url to track the job: http://hadoop103:8088/proxy/application_1724310448291_0001/
2024-08-22 15:47:08,620 INFO mapreduce.Job: Running job: job_1724310448291_0001
2024-08-22 15:47:26,919 INFO mapreduce.Job: Job job_1724310448291_0001 running in uber mode : false
2024-08-22 15:47:26,920 INFO mapreduce.Job:  map 0% reduce 0%
2024-08-22 15:47:39,123 INFO mapreduce.Job: Task Id : attempt_1724310448291_0001_m_000000_0, Status : FAILED
[2024-08-22 15:47:36.966]Container [pid=3230,containerID=container_1724310448291_0001_01_000002] is running 301898240B beyond the 'VIRTUAL' memory limit. Current usage: 177.9 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1724310448291_0001_01_000002 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 3230 3229 3230 3230 (bash) 0 0 9744384 288 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN   -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724310448291_0001/container_1724310448291_0001_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724310448291_0001/container_1724310448291_0001_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.103 41084 attempt_1724310448291_0001_m_000000_0 2 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724310448291_0001/container_1724310448291_0001_01_000002/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724310448291_0001/container_1724310448291_0001_01_000002/stderr
        |- 3240 3230 3230 3230 (java) 389 399 2547011584 45262 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724310448291_0001/container_1724310448291_0001_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724310448291_0001/container_1724310448291_0001_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.103 41084 attempt_1724310448291_0001_m_000000_0 2

[2024-08-22 15:47:37.039]Container killed on request. Exit code is 143
[2024-08-22 15:47:37.046]Container exited with a non-zero exit code 143.

2024-08-22 15:47:47,280 INFO mapreduce.Job: Task Id : attempt_1724310448291_0001_m_000000_1, Status : FAILED
[2024-08-22 15:47:46.025]Container [pid=3284,containerID=container_1724310448291_0001_01_000003] is running 302950912B beyond the 'VIRTUAL' memory limit. Current usage: 287.0 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1724310448291_0001_01_000003 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 3284 3283 3284 3284 (bash) 0 0 9744384 288 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN   -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724310448291_0001/container_1724310448291_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724310448291_0001/container_1724310448291_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.103 41084 attempt_1724310448291_0001_m_000000_1 3 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724310448291_0001/container_1724310448291_0001_01_000003/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724310448291_0001/container_1724310448291_0001_01_000003/stderr
        |- 3294 3284 3284 3284 (java) 407 354 2548064256 73174 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724310448291_0001/container_1724310448291_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724310448291_0001/container_1724310448291_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.103 41084 attempt_1724310448291_0001_m_000000_1 3

[2024-08-22 15:47:46.145]Container killed on request. Exit code is 143
[2024-08-22 15:47:46.164]Container exited with a non-zero exit code 143.

2024-08-22 15:47:59,551 INFO mapreduce.Job: Task Id : attempt_1724310448291_0001_m_000000_2, Status : FAILED
[2024-08-22 15:47:58.499]Container [pid=3523,containerID=container_1724310448291_0001_01_000004] is running 291801600B beyond the 'VIRTUAL' memory limit. Current usage: 108.8 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1724310448291_0001_01_000004 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 3523 3521 3523 3523 (bash) 0 2 9744384 289 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN   -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724310448291_0001/container_1724310448291_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724310448291_0001/container_1724310448291_0001_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.103 41084 attempt_1724310448291_0001_m_000000_2 4 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724310448291_0001/container_1724310448291_0001_01_000004/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724310448291_0001/container_1724310448291_0001_01_000004/stderr
        |- 3532 3523 3523 3523 (java) 188 405 2536914944 27574 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724310448291_0001/container_1724310448291_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724310448291_0001/container_1724310448291_0001_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.103 41084 attempt_1724310448291_0001_m_000000_2 4

[2024-08-22 15:47:58.597]Container killed on request. Exit code is 143
[2024-08-22 15:47:58.636]Container exited with a non-zero exit code 143.

2024-08-22 15:48:06,700 INFO mapreduce.Job:  map 100% reduce 100%
2024-08-22 15:48:07,740 INFO mapreduce.Job: Job job_1724310448291_0001 failed with state FAILED due to: Task failed task_1724310448291_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

2024-08-22 15:48:07,837 INFO mapreduce.Job: Counters: 13
        Job Counters
                Failed map tasks=4
                Killed reduce tasks=1
                Launched map tasks=4
                Other local map tasks=3
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=29799
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=29799
                Total vcore-milliseconds taken by all map tasks=29799
                Total megabyte-milliseconds taken by all map tasks=30514176
        Map-Reduce Framework
                CPU time spent (ms)=0
                Physical memory (bytes) snapshot=0
                Virtual memory (bytes) snapshot=0

解决办法输出重新执行(具体原理不懂)

成功的日志和截屏

[atguigu@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount  /wcinput  /wcoutput
2024-08-22 15:50:03,430 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.188.103:8032
2024-08-22 15:50:04,159 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/atguigu/.staging/job_1724310448291_0002
2024-08-22 15:50:04,289 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 15:50:04,542 INFO input.FileInputFormat: Total input files to process : 1
2024-08-22 15:50:04,582 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 15:50:04,645 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 15:50:04,691 INFO mapreduce.JobSubmitter: number of splits:1
2024-08-22 15:50:04,849 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 15:50:04,917 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1724310448291_0002
2024-08-22 15:50:04,917 INFO mapreduce.JobSubmitter: Executing with tokens: []
2024-08-22 15:50:05,134 INFO conf.Configuration: resource-types.xml not found
2024-08-22 15:50:05,135 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2024-08-22 15:50:05,214 INFO impl.YarnClientImpl: Submitted application application_1724310448291_0002
2024-08-22 15:50:05,285 INFO mapreduce.Job: The url to track the job: http://hadoop103:8088/proxy/application_1724310448291_0002/
2024-08-22 15:50:05,287 INFO mapreduce.Job: Running job: job_1724310448291_0002
2024-08-22 15:50:19,560 INFO mapreduce.Job: Job job_1724310448291_0002 running in uber mode : false
2024-08-22 15:50:19,561 INFO mapreduce.Job:  map 0% reduce 0%
2024-08-22 15:50:24,701 INFO mapreduce.Job:  map 100% reduce 0%
2024-08-22 15:50:30,754 INFO mapreduce.Job:  map 100% reduce 100%
2024-08-22 15:50:30,773 INFO mapreduce.Job: Job job_1724310448291_0002 completed successfully
2024-08-22 15:50:30,897 INFO mapreduce.Job: Counters: 53
        File System Counters
                FILE: Number of bytes read=64
                FILE: Number of bytes written=435581
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=139
                HDFS: Number of bytes written=38
                HDFS: Number of read operations=8
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters
                Launched map tasks=1
                Launched reduce tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=3228
                Total time spent by all reduces in occupied slots (ms)=3229
                Total time spent by all map tasks (ms)=3228
                Total time spent by all reduce tasks (ms)=3229
                Total vcore-milliseconds taken by all map tasks=3228
                Total vcore-milliseconds taken by all reduce tasks=3229
                Total megabyte-milliseconds taken by all map tasks=3305472
                Total megabyte-milliseconds taken by all reduce tasks=3306496
        Map-Reduce Framework
                Map input records=6
                Map output records=7
                Map output bytes=63
                Map output materialized bytes=64
                Input split bytes=103
                Combine input records=7
                Combine output records=5
                Reduce input groups=5
                Reduce shuffle bytes=64
                Reduce input records=5
                Reduce output records=5
                Spilled Records=10
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=141
                CPU time spent (ms)=1470
                Physical memory (bytes) snapshot=532729856
                Virtual memory (bytes) snapshot=5131214848
                Total committed heap usage (bytes)=454033408
                Peak Map Physical memory (bytes)=285954048
                Peak Map Virtual memory (bytes)=2557808640
                Peak Reduce Physical memory (bytes)=246775808
                Peak Reduce Virtual memory (bytes)=2573406208
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=36
        File Output Format Counters
                Bytes Written=38
[atguigu@hadoop102 hadoop-3.1.3]$

结果生成

涉及原理不是很清楚

Hadoop集群启动Hbase的步骤流程
Blackdude的博客
10-19 3602
Hadoop集群启动Hbase的步骤流程 系统版本：centOS7.7 HBASE版本：apache-hbase-1.2.1 HADOOP版本：apache-hadoop-2.7.6 JDK版本：jdk-1.8.0_45 单机安装/伪分布式：1台服务器 全分布式：3台服务器 （关于如何配置hbase，不再叙述，可以查阅资料） 1.启动hdfs start-all.sh 2.启动zookeeper 三台节点同时启动（前提是配置好了 时间同步！） zkServer.sh start 3.启动hbase
hadoop+hbase+zookeeper安装指南
fx20211108的博客
01-20 9002
hadoop、hbase、zookeeper是大数据里面三个重要的产品，关于这三个产品的介绍，各种书籍和网站介绍得很多，在这里不再赘述，仅介绍在ubunt下如何安装和配置这三种产品。 第一步：配置虚拟机为桥接模式 第二步：安装openssh-server，并且登录不需要 /home/jerry/>apt-get install openssh-server #安装openssh-server /home/jerry/>ssh localhost #登录本机，需要密码
hadoop三种启动方式
热门推荐
极客栈
04-25 3万+
 第一种方式 启动：分别启动HDFS和MapReduce 命令如下：start-dfs.sh start-mapreted.sh 命令如下：stop-dfs.sh  stop-mapreted.sh 第二种方式 全部启动或者全部停止 启动： 命令：start-all.sh 启动顺序：NameNode，DateNode，Sec
Hadoop 3.x 启动失败
LZW15082682930的博客
07-11 3162
Hadoop 启动失败
10分钟，手把手教学【正确】还原京东倒计时，初学者必看，简单易懂(1)
2301_79097795的博客
05-09 504
【代码】10分钟，手把手教学【正确】还原京东倒计时，初学者必看，简单易懂(1)
hadoop出现namenode running as process 18472. Stop it first.
weixin_30555753的博客
08-19 691
hadoop出现namenode running as process 18472. Stop it first.等等，类别似几个的出现。 namenode running as process 32972. Stop it first. 127.0.0.1: ssh: connect to host 127.0.0.1 port 22: No error 127.0.0.1: ssh:...
尚硅谷hadoop3.x集群配置笔记及常见错误解决方式
m0_51404279的博客
10-24 5329
1.搭建集群准备工作 总体流程 准备3台客户机（关闭防火墙、静态IP、主机名称） 安装JDK 配置环境变量 安装Hadoop 配置环境变量 配置集群 单点启动 配置ssh 群起并测试集群 一、模板虚拟机的搭建 配置要求：IP地址192.168.10.100**、主机名称hadoop100、内存4G、**硬盘50G（内存硬盘大小根据电脑配置自行更改） 关于是否安装桌面：未安装桌面的版本需要要执行以下命令,用于配置工具包集合，包含ifconfig等命令以及软件...
【Hadoop启动流程全解析】：掌握核心进程、监控及故障诊断
Hadoop启动流程概述 在本章中，我们将带您了解Hadoop这一强大大数据分布式存储与处理框架的启动流程。Hadoop的启动涉及到多个分布式组件和服务的协同工作，其流程设计精心考虑了容错性和高可用性。首先，Hadoop...
HADOOP内容总结（Apache Hadoop）。
weixin_45027256的博客
07-29 746
Hadoop是一个由Apache基金会开发的分布式系统架构，主要解决海量数据的存储和海量数据的计算问题。Hadoop1.xMapReduce计算+资源调度MapReduce计算Yarn资源调度HDFS数据存储HDFS数据存储Common辅助工具Common辅助工具Hadoop Dsitributed File System，简称HDFS，是一个分布式文件系统。主要包含三个节点，分别为NameNode、DataNode和Secondary NameNode。
hadoop一键启动失败
最新发布
02-04
如果之前已经完成过一次完整的安装流程并且进行了初次格式化处理之后再次重做相同的操作，那么就有可能遭遇Datanode和Namenode之间的Cluster ID不匹配现象。此时应当按照官方文档指导清除残留的数据目录再重新开始...
hadoop2.7.3启动脚本流程
07-27
在学习hadoop启动脚本过程中记录的，有一定的参考价值，值得一看！
Hadoop-3.1.3集群搭建详细过程
weixin_49060400的博客
08-16 2130
Hadoop运行环境搭建 1.虚拟机环境准备 1)准备一台模板虚拟机hadoop100，虚拟机配置要求如下： ​ 备注：本文Linux系统环境全部以CentOS-7.5-x86-1804为例说明 模板虚拟机：内存4G，硬盘50G，安装必要环境，为安装hadoop做准备 [root@hadoop100 ~]# yum install -y epel-release [root@hadoop100 ~]# yum install -y psmisc nc net-tools rsync vim lrzsz n
hadoop3学习笔记（零）
NYANYOUNG的博客
03-12 2618
每次重启虚拟机后，打开hadoop的一般流程
Hadoop3 - 基本介绍与使用
小毕超博客
10-03 2401
Hadoop是Apache旗下的一个用java语言实现开源软件框架，是一个开发和运行处理大规模数据的软件平台。允许使用简单的编程模型在大量计算机集群上对大型数据集进行分布式处理。广义上来说，Hadoop通常是指Hadoop生态圈，狭义上说，Hadoop指ApacheHDFS（分布式文件系统）：解决海量数据存储YARN（作业调度和集群资源管理的框架）：解决资源任务调度MAPREDUCE（分布式运算编程框架）：解决海量数据计算。
大数据开发一 Hadoop3.2.1 集群搭建与启动
qq_41515513的博客
10-02 6573
目录 1. 简述 2. 集群搭建与启动 2.1基本环境准备与安装选择 2.2 hadoop的配置文件设置 （重点） 2.3 启动 3. 踩坑记录 4. 结语 1. 简述 作为一名后端工程师，一直对大数据开发有浓厚的兴趣。自主学习已半年有余，特此总结Hadoop3.2.1 在虚拟机(centos6.7)环境下的集群搭建启动流程（从无到有），希望能帮到同在踩坑中的读...
Hadoop 3 端口号的改变及启动Hadoop
oTangBoTu的博客
04-12 2497
Hadoop 3 端口号的改变 分类 应用 Haddop 2.x port Haddop 3 port NNPorts Namenode 8020 9820 NNPorts NN HTTP UI 50070 ...
hadoop集群搭建3之集群启动
Deng0_huakai
11-29 967
前面集群已经成功搭建，现在来尝试启动集群。第一次系统启动的时候，是需要初始化的 启动zookeeper 1.启动zookeeper的命令：./zkServer.sh start|stop|status [hadoop@hadoop001 ~]$3 zkServer.sh start (脚本已经被配置在路径下面了，所以不用再到zookeeper的bin目录下面执行） JMX enabled by d...
hadoop3的单机启动配置
qq_43662627的博客
11-03 707
#下载hadoop的tar包 wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz #解压到指定目录 tar -xzvf hadoop-3.2.1.tar.gz /export/server #修改配置文件 etc/hadoop/core-site.xml <configuration> <property> <na
Hadoop入门-1.配置部署启动（基于hadoop-2.7.3）
上路者的博客
02-21 5763
1.准备 1.下载 hadoop-2.5.2.tar.gz 免安装包到指定的目录 2.执行命令: tar -zxvf hadoop-2.5.2.tar.gz 解压目录 2.配置 1.切换到指定的目录： 2.对相关的配置文件进行配置： core-site.xml 注：此时配置的namenode的地址 hdfs-site.xml 注：配置datanode的数量 slav
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值