部署单节点大数据平台

最新推荐文章于 2024-10-03 20:39:38 发布

余杭杭

最新推荐文章于 2024-10-03 20:39:38 发布

阅读量158

点赞数 1

分类专栏：去搞it吧文章标签： big data hadoop

本文链接：https://blog.csdn.net/u011727208/article/details/119882400

版权

去搞it吧专栏收录该内容

8 篇文章 0 订阅

订阅专栏

环境

主机 hadoop1

系统 centos7

hadoop版本 3.2.2

jdk版本 1.8.0_301

jdk和hadoop安装位置/app/

部署

免密

# 输入四次回车
ssh-keygen -t rsa
cd .ssh/
cat id_rsa.pub > authorized_keys
chmod 600 authorized_keys

解压软件包

tar -zxf hadoop-3.2.2.tar.gz 
tar -zxf jdk-8u301-linux-x64.tar.gz

进入hadoop配置文件目录

cd hadoop-3.2.2/etc/hadoop/

配置hadoop-env.sh

export JAVA_HOME=/app/jdk1.8.0_301

配置core-site.xml

<!-- 制定HDFS的老大（NameNode）的地址 -->
<property>
	<name>fs.defaultFS</name>
	<value>hdfs://hadoop1:9000</value>
</property>
<!-- 指定hadoop运行时产生文件的存储目录 -->
<property>
	<name>hadoop.tmp.dir</name>
	<value>/app/hadoop-3.2.2/tmp</value>
</property>

<property>
	<name>hadoop.proxyuser.hadoop.hosts</name>
	<value>*</value>
</property>
<property>
	<name>hadoop.proxyuser.hadoop.groups</name>
	<value>*</value>
</property>

配置hdfs-site.xml

<property>
	<name>dfs.replication</name>
	<value>1</value>
</property>
<property>
    <name>dfs.namenode.http-address</name>
    <value>hadoop1:50070</value>
</property>

配置mapred-site.xml

<!-- 指定mr运行在yarn上 -->
<property>
	<name>mapreduce.framework.name</name>
	<value>yarn</value>
</property>
<property>
    <name>mapreduce.application.classpath</name>
	<value>
		/app/hadoop-3.2.2/etc/*,
		/app/hadoop-3.2.2/etc/hadoop/*,
		/app/hadoop-3.2.2/lib/*,
		/app/hadoop-3.2.2/share/hadoop/common/*,
		/app/hadoop-3.2.2/share/hadoop/common/lib/*,
		/app/hadoop-3.2.2/share/hadoop/mapreduce/*,
		/app/hadoop-3.2.2/share/hadoop/mapreduce/lib-examples/*,
		/app/hadoop-3.2.2/share/hadoop/hdfs/*,
		/app/hadoop-3.2.2/share/hadoop/hdfs/lib/*,
		/app/hadoop-3.2.2/share/hadoop/yarn/*,
		/app/hadoop-3.2.2/share/hadoop/yarn/lib/*,
	</value>
</property>

配置yarn-site.xml

<!-- 指定YARN的老大（ResourceManager）的地址 -->
<property>
	<name>yarn.resourcemanager.hostname</name>
	<value>hadoop1</value>
</property>
<!-- reducer获取数据的方式 -->
<property>
	<name>yarn.nodemanager.aux-services</name>
	<value>mapreduce_shuffle</value>
</property>
<property>
	<name>yarn.nodemanager.vmem-check-enabled</name>
	<value>false</value>
</property>
<property>
	<name>mapreduce.application.classpath</name>
	<value>
		/app/hadoop-3.2.2/etc/*,
		/app/hadoop-3.2.2/etc/hadoop/*,
		/app/hadoop-3.2.2/lib/*,
		/app/hadoop-3.2.2/share/hadoop/common/*,
		/app/hadoop-3.2.2/share/hadoop/common/lib/*,
		/app/hadoop-3.2.2/share/hadoop/mapreduce/*,
		/app/hadoop-3.2.2/share/hadoop/mapreduce/lib-examples/*,
		/app/hadoop-3.2.2/share/hadoop/hdfs/*,
		/app/hadoop-3.2.2/share/hadoop/hdfs/lib/*,
		/app/hadoop-3.2.2/share/hadoop/yarn/*,
		/app/hadoop-3.2.2/share/hadoop/yarn/lib/*,
	</value>
</property>

配置环境变量

export JAVA_HOME=/app/jdk1.8.0_301
export HADOOP_HOME=/app/hadoop-3.2.2
export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
export CLASSPATH=${CLASSPATH}:$HADOOP_CLASSPATH
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

格式化namenode