大数据环境-hadoop

在这里插入图片描述

基本介绍:

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

准备hadoop-3.1.2.tar.gz:

链接:https://pan.baidu.com/s/1gcw9bM27Xj3AFY_V77-_UA 
提取码:cpc3

hadoop-3.1.2.tar.gz:

链接:https://pan.baidu.com/s/1SSVDpVcAov42ga_yWNS2Yw 
提取码:mboq

1、集群免密登陆配置

vi /etc/hosts 配置主机&主机名

生成公钥:ssh-keygen

查看.ssh文件:

复制公钥到其他节点:

cd~

ssh-copy-id -i .ssh/id_rsa.pub root@slave1

以上步骤重复在几台主机上操作。

注意(需要将公钥追加到authorized_keys文件中去)否则hadoop集群无法正常启动(master:9870无法访问)
命令:cat id_rsa.pub >> authorized_keys

2、环境变量配置(略~参考上一篇)

3、配置文件修改

1)hadoop-env.sh

export HADOOP_OS_TYPE=${HADOOP_OS_TYPE:-$(uname -s)}
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el7_9.x86_64
export HDFS_NAMENODE_USER="root"
export HDFS_DATANODE_USER="root"
export HDFS_SECONDARYNAMENODE_USER="root"
export YARN_RESOURCEMANAGER_USER="root"
export YARN_NODEMANAGER_USER="root"

2)core-site.xml

<configuration>
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
</property>
<property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop-3.1.2/data/tmp</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>

<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>

<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
</configuration>

3)hdfs-site.xml

<configuration>
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
</property>
<property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop-3.1.2/data/tmp</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>

<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>

<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
</configuration>
 

4)yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
         <name>yarn.nodemanager.aux-services</name>
         <value>mapreduce_shuffle</value>
</property>
<!-- 指定YARN的ResourceManager的地址 -->
<property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
</property>
<property>
        <name>yarn.application.classpath</name>
        <value>/usr/local/hadoop-3.1.2/etc/hadoop:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/*:/usr/local/hadoop-3.1.2/share/hadoop/common/*:/usr/local/hadoop-3.1.2/share/hadoop/hdfs:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/*:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/*:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/*:/usr/local/hadoop-3.1.2/share/hadoop/yarn:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/*:/usr/local/hadoop-3.1.2/share/hadoop/yarn/*</value>
</property>
</configuration>
 

5)mapred-site.xml

<configuration>
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>
<property>
        <name>mapreduce.jobhistory.address</name>
        <value>master:10020</value>
</property>
<property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>master:19888</value>
</property>

</configuration>

配置从节点workers文件,添加从节点服务器:

分发到其他节点,并给每个节点配置环境变量

 scp -r /usr/local/hadoop-3.1.2 slave1:/usr/local/hadoop-3.1.2/

启动hadoop(提前关闭防火墙)

查看防火墙状态:firewall-cmd --state

关闭防火墙: systemctl stop firewalld.service

首次启动Hadoop集群格式化和初始化集群

hdfs --daemon start journalnode

hdfs namenode -format

start-dfs.sh

./stop-all.sh

hdfs zkfc -formatZK

正式启动:

hdfs --workers --daemon start journalnode

./start-all.sh

http://master:8088/

 http://master:9870/

 

 

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值