Hadoop 环境变量配置:
HADOOP_HOME:D:\ProgramData\BigData\Hadoop\hadoop-2.7.4
Path:%HADOOP_HOME%\bin
Spark 环境变量配置:
SPARK_HOME:D:\ProgramData\BigData\Spark\spark-2.2.0
Path:%SPARK_HOME%\bin
修改hadoop配置文件:
[1].编辑D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop,下的core-site.xml文件:
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/D:/ProgramData/BigData/Hadoop/hadoop-2.7.4/workplace/tmp</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/D:/ProgramData/BigData/Hadoop/hadoop-2.7.4/workplace/name</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:8084/Hadoop</value>
</property>
</configuration>
[2].编辑“D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop”目录下的mapred-site.xml(没有就将mapred-site.xml.template重命名为mapred-site.xml)文件
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://localhost:8085/Hadoop</value>
</property>
</configuration>
[3].编辑“D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop”目录下的hdfs-site.xml文件:
<configuration>
<!-- 这个参数设置为1,因为是单机版hadoop -->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/D:/ProgramData/BigData/Hadoop/hadoop-2.7.4/workplace/data</value>
</property>
</configuration>
[4].编辑“D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop”目录下的yarn-site.xml文件:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
[5].编辑“D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop”目录下的hadoop-env.cmd文件,将JAVA_HOME用 @rem注释掉,编辑为JAVA_HOME的路径:
set JAVA_HOME=D:/ProgramLanguage/Java/jdk/jdk1.8.0_144
[6].下载到的hadooponwindows-master.zip,解压,将bin目录(包含以下.dll和.exe文件)文件替换原来hadoop目录下的bin目录
[7].
hadoop fs -mkdir hdfs://localhost:8087/user
hadoop fs -mkdir hdfs://localhost:8087/user/wcinput
hadoop fs -put D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\workplace\data\file1.txt hdfs://localhost:8087/user/wcinput
hadoop fs -put D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\workplace\data\file2.txt hdfs://localhost:8087/user/wcinput
hadoop fs -ls hdfs://localhost:8087/user/wcinput
HADOOP_HOME:D:\ProgramData\BigData\Hadoop\hadoop-2.7.4
Path:%HADOOP_HOME%\bin
Spark 环境变量配置:
SPARK_HOME:D:\ProgramData\BigData\Spark\spark-2.2.0
Path:%SPARK_HOME%\bin
修改hadoop配置文件:
[1].编辑D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop,下的core-site.xml文件:
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/D:/ProgramData/BigData/Hadoop/hadoop-2.7.4/workplace/tmp</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/D:/ProgramData/BigData/Hadoop/hadoop-2.7.4/workplace/name</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:8084/Hadoop</value>
</property>
</configuration>
[2].编辑“D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop”目录下的mapred-site.xml(没有就将mapred-site.xml.template重命名为mapred-site.xml)文件
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://localhost:8085/Hadoop</value>
</property>
</configuration>
[3].编辑“D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop”目录下的hdfs-site.xml文件:
<configuration>
<!-- 这个参数设置为1,因为是单机版hadoop -->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/D:/ProgramData/BigData/Hadoop/hadoop-2.7.4/workplace/data</value>
</property>
</configuration>
[4].编辑“D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop”目录下的yarn-site.xml文件:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
[5].编辑“D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop”目录下的hadoop-env.cmd文件,将JAVA_HOME用 @rem注释掉,编辑为JAVA_HOME的路径:
set JAVA_HOME=D:/ProgramLanguage/Java/jdk/jdk1.8.0_144
[6].下载到的hadooponwindows-master.zip,解压,将bin目录(包含以下.dll和.exe文件)文件替换原来hadoop目录下的bin目录
[7].
hadoop fs -mkdir hdfs://localhost:8087/user
hadoop fs -mkdir hdfs://localhost:8087/user/wcinput
hadoop fs -put D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\workplace\data\file1.txt hdfs://localhost:8087/user/wcinput
hadoop fs -put D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\workplace\data\file2.txt hdfs://localhost:8087/user/wcinput
hadoop fs -ls hdfs://localhost:8087/user/wcinput