大数据系列（3）——linux的hadoop安装

最新推荐文章于 2023-02-27 16:04:52 发布

EVAO

最新推荐文章于 2023-02-27 16:04:52 发布

阅读量148

点赞数

分类专栏：大数据

本文链接：https://blog.csdn.net/qq_31683033/article/details/91989978

版权

大数据专栏收录该内容

17 篇文章 0 订阅

订阅专栏

1. hadoop的安装类型

1.1 hadoop单机版

1.2 hadoop伪分布式版

1.3 hadoop完全分布式版

2. hadoop单机版的安装(准备工作)

2.1 关闭selinux

vim /etc/selinux/config
	SELINUX=disabled

2.2 关闭图形接口

vim /etc/inittab
	id:3:initdefault:

2.3 关闭防火墙

service iptables stop  #关闭防火墙服务
chkconfig  iptables  off #关闭开机自动启动

2.4 修改主机名称

vim /etc/sysconfig/network
    NETWORKING=yes
    HOSTNAME=uplooking01

2.5 安装jdk

tar -zxvf jdk-8u181-linux-x64.tar.gz -C /opt/  #解压jdk到/opt目录
mv jdk1.8.0_181/ jdk  #重命名

2.6 配置环境变量

vim /etc/profile
    export JAVA_HOME=/opt/jdk
    export PATH=$PATH:$JAVA_HOME/bin

2.7 使环境变量立刻生效

source /etc/profile

2.8 修改主机映射

vim /etc/hosts
	10.10.10.11 uplooking01

3. hadoop单机版的安装(开始安装)

3.1 上传hadoop的软件包

https://archive.apache.org/dist/hadoop/common/hadoop-2.6.4/

hadoop-2.6.4-bin_x64.tar.gz

3.2 解压hadoop

tar -zxvf hadoop-2.6.4-bin_x64.tar.gz -C /opt/

3.3 重命名hadoop

mv hadoop-2.6.4/ hadoop

3.4 删除hadoop中的*.cmd

 rm -rf /opt/hadoop/bin/*.cmd /opt/hadoop/sbin/*.cmd  /opt/hadoop/etc/hadoop/*.cmd

3.5 配置环境变量

#配置hadoop的环境变量
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

3.6 hadoop-env.sh

export JAVA_HOME=/opt/jdk

3.7 yarn-env.sh

export JAVA_HOME=/opt/jdk

3.8 mapred-env.sh

export JAVA_HOME=/opt/jdk

3.9 core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://uplooking01</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:///opt/hadoop-repo/tmp</value>
    </property>
</configuration>

3.10 hdfs-site.xml

<configuration>
    <property>  
        <name>dfs.namenode.name.dir</name>  
        <value>file:///opt/hadoop-repo/name</value>  
    </property>

    <property> 
        <name>dfs.datanode.data.dir</name>  
        <value>file:///opt/hadoop-repo/data</value>  
    </property>
    <property>
        <name>dfs.namenode.checkpoint.dir</name>
        <value>file:///opt/hadoop-repo/secondary</value>
    </property>
    <!-- secondaryName http地址 -->
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>uplooking01:9001</value>
    </property>
    <!-- 数据备份数量-->
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <!-- 运行通过web访问hdfs-->
    <property> 
        <name>dfs.webhdfs.enabled</name>  
        <value>true</value>  
    </property>
    <!-- 剔除权限控制-->
    <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property> 
</configuration>

3.11 mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property> 
    <!-- 历史job的访问地址-->
    <property>  
        <name>mapreduce.jobhistory.address</name>  
        <value>uplooking01:10020</value>  
    </property>
    <!-- 历史job的访问web地址-->
    <property>  
        <name>mapreduce.jobhistory.webapp.address</name>  
        <value>uplooking01:19888</value>  
    </property>
    <property>
        <name>mapreduce.map.log.level</name>
        <value>INFO</value>
    </property>
    <property>
        <name>mapreduce.reduce.log.level</name>
        <value>INFO</value>
    </property>
</configuration>

3.12 yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>uplooking01</value>
    </property> 
    <property>  
        <name>yarn.resourcemanager.address</name>  
        <value>uplooking01:8032</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.scheduler.address</name>  
        <value>uplooking01:8030</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.resource-tracker.address</name>  
        <value>uplooking01:8031</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.admin.address</name>  
        <value>uplooking01:8033</value>  
    </property>
    <property> 
        <name>yarn.resourcemanager.webapp.address</name>  
        <value>uplooking01:8088</value>  
    </property>
    <property>
        <name>yarn.log-aggregation-enable</name>  
        <value>true</value>  
    </property>
</configuration>

3.13 创建目录

mkdir /opt/hadoop-repo
mkdir /opt/hadoop-repo/name
mkdir /opt/hadoop-repo/data
mkdir /opt/hadoop-repo/tmp
mkdir /opt/hadoop-repo/secondary

3.14 配置免密码登录

ssh-keygen -t "rsa"  #生成密钥对
ssh-copy-id uplooking01

3.15 格式化文件系统(谨慎操作)

hadoop namenode -format

3.16 启动hadoop

start-all.sh  #启动hdfs和yarn

3.17 访问hadoop

4. hdfs(分布式文件系统)

4.1 hdfs的特点

分布式
- 数据量越来越多，在一个操作系统管辖的范围存不下了，那么就分配到更多的操作系统管理的磁盘中，但是不方便管理和维护，因此迫切需要一种系统来管理多台机器上的文件，这就是分布式文件管理系统
高可用
- 副本机制
通透性
- 用户通过网络访问hdfs上的文件就像访问本地文件系统一样
容错机制

4.2 hdfs的架构

4.2.1 NameNode

文件系统的管理节点
它维护着整个文件系统的文件目录树，文件/目录的元信息和每个文件对应的数据块列表
接收用户的操作请求

4.2.2 DataNode

提供真实文件数据的存储服务 (存储的是Block)

4.2.3 Block

最基本的存储单位
在hadoop1.x的版本中一个block的大小为64mb,在hadoop2.x的版本中一个block大大小为128mb
如果一个文件小于一个数据块的大小，并不占用整个数据块存储空间

5. hadoop完全分布式版的安装(准备工作)

5.1 关闭selinux

vim /etc/selinux/config
	SELINUX=disabled

5.2 关闭图形接口

vim /etc/inittab
	id:3:initdefault:

5.3 关闭防火墙

service iptables stop  #关闭防火墙服务
chkconfig  iptables  off #关闭开机自动启动

5.4 修改主机名称

vim /etc/sysconfig/network
    NETWORKING=yes
    HOSTNAME=uplooking01

5.5 安装jdk

tar -zxvf jdk-8u181-linux-x64.tar.gz -C /opt/  #解压jdk到/opt目录
mv jdk1.8.0_181/ jdk  #重命名

5.6 配置环境变量

vim /etc/profile
    export JAVA_HOME=/opt/jdk
    export PATH=$PATH:$JAVA_HOME/bin

5.7 使环境变量立刻生效

source /etc/profile

5.8 修改主机映射

vim /etc/hosts
	10.10.10.11 uplooking01

5.9 修改root用户密码

passwd root

6. hadoop完全分布式版的安装(开始安装)

6.1 上传hadoop的软件包

https://archive.apache.org/dist/hadoop/common/hadoop-2.6.4/

hadoop-2.6.4-bin_x64.tar.gz

6.2 解压hadoop

tar -zxvf hadoop-2.6.4-bin_x64.tar.gz -C /opt/

6.3 重命名hadoop

mv hadoop-2.6.4/ hadoop

6.4 删除hadoop中的*.cmd

 rm -rf /opt/hadoop/bin/*.cmd /opt/hadoop/sbin/*.cmd  /opt/hadoop/etc/hadoop/*.cmd

6.5 配置环境变量

#配置hadoop的环境变量
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

6.6 hadoop-env.sh

export JAVA_HOME=/opt/jdk

6.7 yarn-env.sh

export JAVA_HOME=/opt/jdk

6.8 mapred-env.sh

export JAVA_HOME=/opt/jdk

6.9 core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://uplooking01</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/hadoop-repo/tmp</value>
    </property>
</configuration>

6.10 hdfs-site.xml

<configuration>
	<property>  
		<name>dfs.namenode.name.dir</name>  
		<value>/opt/hadoop-repo/name</value>  
	</property>
	
	<property> 
		<name>dfs.datanode.data.dir</name>  
		<value>/opt/hadoop-repo/data</value>  
	</property>
	<property>
		<name>dfs.namenode.checkpoint.dir</name>
		<value>/opt/hadoop-repo/secondary</value>
	</property>
	<!-- secondaryName http地址 -->
	<property>
		<name>dfs.namenode.secondary.http-address</name>
		<value>uplooking02:9001</value>
	</property>
	<!-- 数据备份数量-->
	<property>
		<name>dfs.replication</name>
		<value>3</value>
	</property>
	<!-- 运行通过web访问hdfs-->
	<property> 
		<name>dfs.webhdfs.enabled</name>  
		<value>true</value>  
	</property>
	<!-- 剔除权限控制-->
	<property>
		<name>dfs.permissions</name>
		<value>false</value>
	</property> 
</configuration>

6.11 mapred-site.xml

<configuration>
	<property>
		<name>mapreduce.framework.name</name>
		<value>yarn</value>
	</property> 
	<!-- 历史job的访问地址-->
	<property>  
		<name>mapreduce.jobhistory.address</name>  
		<value>uplooking02:10020</value>  
	</property>
	<!-- 历史job的访问web地址-->
	<property>  
		<name>mapreduce.jobhistory.webapp.address</name>  
		<value>uplooking02:19888</value>  
	</property>
	<property>
		<name>mapreduce.map.log.level</name>
		<value>INFO</value>
	</property>
	<property>
		<name>mapreduce.reduce.log.level</name>
		<value>INFO</value>
	</property>
</configuration>

6.12 yarn-site.xml

<configuration>
	<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
	</property>
	<property>
		<name>yarn.resourcemanager.hostname</name>
		<value>uplooking02</value>
	</property> 
	<property>  
		<name>yarn.resourcemanager.address</name>  
		<value>uplooking02:8032</value>  
	</property>  
	<property>  
		<name>yarn.resourcemanager.scheduler.address</name>  
		<value>uplooking02:8030</value>  
	</property>  
	<property>  
		<name>yarn.resourcemanager.resource-tracker.address</name>  
		<value>uplooking02:8031</value>  
	</property>  
	<property>  
		<name>yarn.resourcemanager.admin.address</name>  
		<value>uplooking02:8033</value>  
	</property>
	<property> 
		<name>yarn.resourcemanager.webapp.address</name>  
		<value>uplooking02:8088</value>  
	</property>
	<property>
		<name>yarn.log-aggregation-enable</name>  
		<value>true</value>  
	</property>
</configuration>

6.13 删除repo目录

删除uplooking01上的/opt/hadoop-repo

6.14 克隆主机

uplooking01 nn
uplooking02 2nn rm
uplooking03 dn nm
uplooking04 dn nm
uplooking05 dn nm

6.15 解决克隆主机后网卡出现eth1的问题

rm -rf /etc/udev/rules.d/70-persistent-net.rules #删除之后重启即可

6.16 建立主机之间的映射

[root@uplooking01] vim /etc/hosts
	10.10.10.11 uplooking01
    10.10.10.12 uplooking02
    10.10.10.13 uplooking03
    10.10.10.14 uplooking04
    10.10.10.15 uplooking05

6.17 分发文件

[root@uplooking01]  
	scp /etc/hosts  10.10.10.12:/etc/
	scp /etc/hosts  10.10.10.13:/etc/
	scp /etc/hosts  10.10.10.14:/etc/
	scp /etc/hosts  10.10.10.15:/etc/

6.18 编辑slaves文件

[root@uplooking01]  
	vim /opt/hadoop/etc/hadoop/slaves
			uplooking03
			uplooking04
			uplooking05

6.19 分发slaves文件

[root@uplooking01]  
	 scp /opt/hadoop/etc/hadoop/slaves  uplooking02:/opt/hadoop/etc/hadoop/
	 scp /opt/hadoop/etc/hadoop/slaves  uplooking03:/opt/hadoop/etc/hadoop/
	 scp /opt/hadoop/etc/hadoop/slaves  uplooking04:/opt/hadoop/etc/hadoop/
	 scp /opt/hadoop/etc/hadoop/slaves  uplooking05:/opt/hadoop/etc/hadoop/

6.20 开启hadoop集群

[root@uplooking01] 
	start-all.sh

ps:此命令有个小bug就是不能启动resourcemanager我们需要在==uplooking02==上手动启动resourcemanager

yarn-daemon.sh start resourcemanager

6.21 验证集群

EVAO

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
大数据系列（3）——linux的hadoop安装

1. hadoop的安装类型1.1 hadoop单机版1.2 hadoop伪分布式版1.3 hadoop完全分布式版2. hadoop单机版的安装(准备工作)2.1 关闭selinuxvim /etc/selinux/config SELINUX=disabled2.2 关闭图形接口vim /etc/inittab id:3:initdefault:2.3 关闭防火墙s...
复制链接

扫一扫

专栏目录