hadoop 全分布式配置（实机配置）

Bernard5

已于 2022-07-03 16:06:56 修改

阅读量743

点赞数 1

分类专栏：配环境/安装文章标签： hadoop big data

于 2021-09-18 12:20:29 首次发布

本文禁止任何形式的转载，作者保有所有权利

本文链接：https://blog.csdn.net/sinat_27873735/article/details/120362580

版权

配环境/安装专栏收录该内容

30 篇文章 0 订阅

订阅专栏

查看ip

enp0s31f6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.5  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::b914:1d47:742a:cbb6  prefixlen 64  scopeid 0x20<link>
        ether 30:9c:23:f9:4c:d5  txqueuelen 1000  (Ethernet)
        RX packets 2979086  bytes 4322586157 (4.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1402179  bytes 115472378 (110.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  memory 0xf7100000-f7120000

wlp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.3  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::5d8f:7994:66c9:d0d6  prefixlen 64  scopeid 0x20<link>
        ether 5c:93:a2:bb:53:e9  txqueuelen 1000  (Ethernet)
        RX packets 203814  bytes 230785909 (230.7 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 121152  bytes 15864087 (15.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

配置host

sudo vi /etc/hosts

192.168.0.5 master
192.168.0.3 slave

hadoop@aqua:~$ ping master
PING master (192.168.0.5) 56(84) bytes of data.
64 bytes from master (192.168.0.5): icmp_seq=1 ttl=64 time=1.70 ms
64 bytes from master (192.168.0.5): icmp_seq=2 ttl=64 time=1.18 ms
64 bytes from master (192.168.0.5): icmp_seq=3 ttl=64 time=1.29 ms
64 bytes from master (192.168.0.5): icmp_seq=4 ttl=64 time=1.06 ms
^C
--- master ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 1.063/1.306/1.697/0.239 ms

需要配置master到slave上的免密码登陆，否则需要调整hadoop-env.sh

ssh localhost # 如果已经存在./ssh

配置

cd hadoop-3.3.1/etc/hadoop/

worker

vi workers

slave

core-site.xml

vi core-site.xml

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://Master:9000</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>file:/opt/hadoop-3.3.1/tmp</value>
                <description>Abase for other temporary directories.</description>
        </property>
</configuration>

hdfs-site.xml

vi hdfs-site.xml

<configuration>
        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>master:50090</value>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:/opt/hadoop-3.3.1/tmp/dfs/name</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>file:/opt/hadoop-3.3.1/tmp/dfs/data</value>
        </property>
</configuration>

mapred-site.xml

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>master:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>master:19888</value>
        </property>
        <property>
                <name>yarn.app.mapreduce.am.env</name>
                <value>HADOOP_MAPRED_HOME=/opt/hadoop-3.3.1</value>
        </property>
        <property>
                <name>mapreduce.map.env</name>
                <value>HADOOP_MAPRED_HOME=/opt/hadoop-3.3.1</value>
        </property>
        <property>
                <name>mapreduce.reduce.env</name>
                <value>HADOOP_MAPRED_HOME=/opt/hadoop-3.3.1</value>
        </property> 
</configuration>

yarn-site.xml

<configuration>
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>master</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
</configuration>

启动

启动之前要删除之前的运行结果的文件，如伪分布式运行结果。
主从节点之间配置文件需要相同，但环境变量如java可以不同。

hdfs namenode -format
start-dfs.sh
start-yarn.sh
mr-jobhistory-daemon.sh start historyserver # 此处警告

[bernard@fedora hadoop-3.3.1]$ hdfs namenode -format
WARNING: /opt/hadoop-3.3.1/logs does not exist. Creating.
2021-09-18 12:06:53,252 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = master/192.168.0.5
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 3.3.1
STARTUP_MSG:   # 省略
STARTUP_MSG:   build = https://github.com/apache/hadoop.git -r a3b9c37a397ad4188041dd80621bdeefc46885f2; compiled by 'ubuntu' on 2021-06-15T05:13Z
STARTUP_MSG:   java = 1.8.0_301
************************************************************/
2021-09-18 12:06:53,261 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2021-09-18 12:06:53,328 INFO namenode.NameNode: createNameNode [-format]
2021-09-18 12:06:53,700 INFO namenode.NameNode: Formatting using clusterid: CID-027f3e31-27e3-43a9-82e2-150e1e8981fd
2021-09-18 12:06:53,726 INFO namenode.FSEditLog: Edit logging is async:true
2021-09-18 12:06:53,746 INFO namenode.FSNamesystem: KeyProvider: null
2021-09-18 12:06:53,748 INFO namenode.FSNamesystem: fsLock is fair: true
2021-09-18 12:06:53,748 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2021-09-18 12:06:53,753 INFO namenode.FSNamesystem: fsOwner                = bernard (auth:SIMPLE)
2021-09-18 12:06:53,753 INFO namenode.FSNamesystem: supergroup             = supergroup
2021-09-18 12:06:53,753 INFO namenode.FSNamesystem: isPermissionEnabled    = true
2021-09-18 12:06:53,753 INFO namenode.FSNamesystem: isStoragePolicyEnabled = true
2021-09-18 12:06:53,753 INFO namenode.FSNamesystem: HA Enabled: false
2021-09-18 12:06:53,788 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2021-09-18 12:06:53,795 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
2021-09-18 12:06:53,795 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2021-09-18 12:06:53,797 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2021-09-18 12:06:53,797 INFO blockmanagement.BlockManager: The block deletion will start around 2021 九月 18 12:06:53
2021-09-18 12:06:53,798 INFO util.GSet: Computing capacity for map BlocksMap
2021-09-18 12:06:53,798 INFO util.GSet: VM type       = 64-bit
2021-09-18 12:06:53,800 INFO util.GSet: 2.0% max memory 1.7 GB = 35.0 MB
2021-09-18 12:06:53,800 INFO util.GSet: capacity      = 2^22 = 4194304 entries
2021-09-18 12:06:53,806 INFO blockmanagement.BlockManager: Storage policy satisfier is disabled
2021-09-18 12:06:53,806 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
2021-09-18 12:06:53,830 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.999
2021-09-18 12:06:53,830 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2021-09-18 12:06:53,830 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
2021-09-18 12:06:53,831 INFO blockmanagement.BlockManager: defaultReplication         = 1
2021-09-18 12:06:53,831 INFO blockmanagement.BlockManager: maxReplication             = 512
2021-09-18 12:06:53,831 INFO blockmanagement.BlockManager: minReplication             = 1
2021-09-18 12:06:53,831 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
2021-09-18 12:06:53,831 INFO blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms
2021-09-18 12:06:53,831 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
2021-09-18 12:06:53,831 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
2021-09-18 12:06:53,846 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911
2021-09-18 12:06:53,846 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215
2021-09-18 12:06:53,846 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215
2021-09-18 12:06:53,846 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215
2021-09-18 12:06:53,855 INFO util.GSet: Computing capacity for map INodeMap
2021-09-18 12:06:53,855 INFO util.GSet: VM type       = 64-bit
2021-09-18 12:06:53,855 INFO util.GSet: 1.0% max memory 1.7 GB = 17.5 MB
2021-09-18 12:06:53,855 INFO util.GSet: capacity      = 2^21 = 2097152 entries
2021-09-18 12:06:53,856 INFO namenode.FSDirectory: ACLs enabled? true
2021-09-18 12:06:53,856 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
2021-09-18 12:06:53,856 INFO namenode.FSDirectory: XAttrs enabled? true
2021-09-18 12:06:53,857 INFO namenode.NameNode: Caching file names occurring more than 10 times
2021-09-18 12:06:53,862 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
2021-09-18 12:06:53,863 INFO snapshot.SnapshotManager: SkipList is disabled
2021-09-18 12:06:53,867 INFO util.GSet: Computing capacity for map cachedBlocks
2021-09-18 12:06:53,867 INFO util.GSet: VM type       = 64-bit
2021-09-18 12:06:53,867 INFO util.GSet: 0.25% max memory 1.7 GB = 4.4 MB
2021-09-18 12:06:53,867 INFO util.GSet: capacity      = 2^19 = 524288 entries
2021-09-18 12:06:53,873 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2021-09-18 12:06:53,873 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2021-09-18 12:06:53,873 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2021-09-18 12:06:53,876 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
2021-09-18 12:06:53,877 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2021-09-18 12:06:53,878 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2021-09-18 12:06:53,878 INFO util.GSet: VM type       = 64-bit
2021-09-18 12:06:53,878 INFO util.GSet: 0.029999999329447746% max memory 1.7 GB = 538.1 KB
2021-09-18 12:06:53,878 INFO util.GSet: capacity      = 2^16 = 65536 entries
2021-09-18 12:06:53,900 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1916113005-192.168.0.5-1631938013893
2021-09-18 12:06:53,910 INFO common.Storage: Storage directory /opt/hadoop-3.3.1/tmp/dfs/name has been successfully formatted.
2021-09-18 12:06:53,928 INFO namenode.FSImageFormatProtobuf: Saving image file /opt/hadoop-3.3.1/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2021-09-18 12:06:54,005 INFO namenode.FSImageFormatProtobuf: Image file /opt/hadoop-3.3.1/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 402 bytes saved in 0 seconds .
2021-09-18 12:06:54,015 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2021-09-18 12:06:54,033 INFO namenode.FSNamesystem: Stopping services started for active state
2021-09-18 12:06:54,033 INFO namenode.FSNamesystem: Stopping services started for standby state
2021-09-18 12:06:54,035 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2021-09-18 12:06:54,035 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.0.5
************************************************************/


[bernard@fedora ~]$ start-dfs.sh
Starting namenodes on [master]
master: namenode is running as process 57862.  Stop it first and ensure /tmp/hadoop-bernard-namenode.pid file is empty before retry.
Starting datanodes
slave: ERROR: Cannot execute /opt/software/hadoop-3.3.1/libexec/hdfs-config.sh.
Starting secondary namenodes [master]
master: secondarynamenode is running as process 58130.  Stop it first and ensure /tmp/hadoop-bernard-secondarynamenode.pid file is empty before retry.
[bernard@fedora ~]$ start-dfs.sh
Starting namenodes on [master]
master: namenode is running as process 57862.  Stop it first and ensure /tmp/hadoop-bernard-namenode.pid file is empty before retry.
Starting datanodes
Starting secondary namenodes [master]
master: secondarynamenode is running as process 58130.  Stop it first and ensure /tmp/hadoop-bernard-secondarynamenode.pid file is empty before retry.
[bernard@fedora ~]$ jps
58130 SecondaryNameNode
57862 NameNode
42602 ResourceManager
59310 Jps
[bernard@fedora ~]$ start-yarn.sh
Starting resourcemanager
resourcemanager is running as process 42602.  Stop it first and ensure /tmp/hadoop-bernard-resourcemanager.pid file is empty before retry.
Starting nodemanagers
[bernard@fedora ~]$ jps
58130 SecondaryNameNode
59492 Jps
57862 NameNode
42602 ResourceManager
[bernard@fedora ~]$ mr-jobhistory-daemon.sh start historyserver
WARNING: Use of this script to start the MR JobHistory daemon is deprecated.
WARNING: Attempting to execute replacement "mapred --daemon start" instead.
[bernard@fedora ~]$ jps
59553 JobHistoryServer
58130 SecondaryNameNode
59620 Jps
57862 NameNode
42602 ResourceManager

在这里插入图片描述

reference

http://dblab.xmu.edu.cn/blog/2775-2/

遗留问题

hadoop用户权限不够

[hadoop@fedora hadoop-3.3.1]$ ls
bin  include  libexec         licenses-binary  logs           NOTICE.txt  sbin   tmp
etc  lib      LICENSE-binary  LICENSE.txt      NOTICE-binary  README.txt  share
[hadoop@fedora hadoop-3.3.1]$ sudo rm -r logs/
[hadoop@fedora hadoop-3.3.1]$ sudo rm -r tmp/
[hadoop@fedora hadoop-3.3.1]$ hdfs namenode -format
WARNING: /opt/hadoop-3.3.1/logs does not exist. Creating.
mkdir: 无法创建目录 “/opt/hadoop-3.3.1/logs”: 权限不够
ERROR: Unable to create /opt/hadoop-3.3.1/logs. Aborting.
[hadoop@fedora hadoop-3.3.1]$ hdfs namenode -format
WARNING: /opt/hadoop-3.3.1/logs does not exist. Creating.
mkdir: 无法创建目录 “/opt/hadoop-3.3.1/logs”: 权限不够
ERROR: Unable to create /opt/hadoop-3.3.1/logs. Aborting.

创建方式

sudo useradd -m hadoop -s /bin/bash
sudo passwd hadoop
sudo adduser hadoop sudo # 此处报错，遂手动调整/etc/sudoer

Bernard5

关注

1
点赞
踩
0

收藏

觉得还不错? 一键收藏
打赏
0
评论
hadoop 全分布式配置（实机配置）

查看ipenp0s31f6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.0.5 netmask 255.255.255.0 broadcast 192.168.0.255 inet6 fe80::b914:1d47:742a:cbb6 prefixlen 64 scopeid 0x20<link> ether 30:9c:23:f9:4c
复制链接

扫一扫