基础架构系列篇-基于win10安装SPARK3与HADOOP环境

基础架构系列篇-基于win10安装SPARK3与HADOOP环境(这里版本是 spark-3.1.2 hadoop-3.2.0)

1. 下载安装spark (这里版本是 spark-3.1.2-bin-hadoop3.2.tgz)

地址: http://spark.apache.org/downloads.html
https://downloads.apache.org/spark
https://archive.apache.org/dist/spark
解压并设置环境变量SPARK_HOME
增加PATH %SPARK_HOME%\bin %SPARK_HOME%\sbin
打开POWERSHELL测试spark-shell

2.下载安装hadoop (按SPARK版本选择)

地址: https://archive.apache.org/dist/hadoop/common/hadoop-3.2.0/
http://archive.apache.org/dist/hadoop/core/
解压并设置环境变量HADOOP_HOME
增加PATH %HADOOP_HOME%\bin %HADOOP_HOME%\sbin

下载WINUTILS (按HADOOP版本选择)
地址: https://github.com/cdarlint/winutils
拷贝WINUTILS的BIN内容放HADOOP解压的BIN路径
(可不拷贝WINUTILS的BIN的hadoop.dll放c盘windows/sysyem32下)
打开POWERSHELL测试hadoop version

设置hadoop参数 路径etc/hadoop
hadoop-env.cmd (可不改)
@set JAVA_HOME=%JAVA_HOME%
set JAVA_HOME=安装的JDK路径

core-site.xml

<configuration> 
<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:8900</value>
</property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/F:/SW/BIGdata/hadoop/hadoop-3.2.0/data/tmp/${user.name}</value>
    <description>A base for other temporary directories.</description>
</property>
</configuration>

hdfs-site.xml

<configuration>
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
<property>
    <name>dfs.permissions</name>
    <value>false</value>
</property>
<property>
    <name>dfs.namenode.name.dir</name>
    <value>/F:/SW/BIGdata/hadoop/hadoop-3.2.0/data/dfs/namenode</value>
</property>
<property>
    <name>dfs.datanode.data.dir</name>
    <value>/F:/SW/BIGdata/hadoop/hadoop-3.2.0/data/dfs/datanode</value>
</property>
<property>
    <name>fs.checkpoint.dir</name>
    <value>/F:/SW/BIGdata/hadoop/hadoop-3.2.0/data/dfs/snn</value>
</property>
<property>
    <name>fs.checkpoint.edits.dir</name>
    <value>/F:/SW/BIGdata/hadoop/hadoop-3.2.0/data/dfs/snn</value>
</property>
</configuration>

yarn-site.xml

<configuration> 
 <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

mapred-site.xml

<configuration> 
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
</configuration>

格式化NameNode
hdfs namenode -format
(版本hadoop-3.2.1异常ERROR namenode.NameNode: Failed to start namenode.
java.lang.UnsupportedOperationException at java.nio.file.Files.setPosixFilePermissions(Files.java:2044))
修改用hadoop-3.2.0版本

启动Hadoop服务
执行start-all
停止stop-all

3. 测试查看

yarn：http://127.0.0.1:8088/cluster
hdfs：http://localhost:9870