Hadoop+Spark 环境搭建

最新推荐文章于 2024-06-20 23:10:26 发布

weixin_34050427

最新推荐文章于 2024-06-20 23:10:26 发布

阅读量82

点赞数

文章标签：大数据 python java

原文链接：https://my.oschina.net/2273990296/blog/1524434

版权

2019独角兽企业重金招聘Python工程师标准>>>

Hadoop 环境变量配置： HADOOP_HOME:D:\ProgramData\BigData\Hadoop\hadoop-2.7.4 Path：%HADOOP_HOME%\bin

Spark 环境变量配置： SPARK_HOME：D:\ProgramData\BigData\Spark\spark-2.2.0 Path：%SPARK_HOME%\bin

修改hadoop配置文件： [1].编辑D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop，下的core-site.xml文件： <configuration> <property> <name>hadoop.tmp.dir</name> <value>/D:/ProgramData/BigData/Hadoop/hadoop-2.7.4/workplace/tmp</value> </property> <property> <name>dfs.name.dir</name> <value>/D:/ProgramData/BigData/Hadoop/hadoop-2.7.4/workplace/name</value> </property> <property> <name>fs.default.name</name> <value>hdfs://localhost:8084/Hadoop</value> </property> </configuration>

[2].编辑“D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop”目录下的mapred-site.xml(没有就将mapred-site.xml.template重命名为mapred-site.xml)文件 <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapred.job.tracker</name> <value>hdfs://localhost:8085/Hadoop</value> </property> </configuration>

[3].编辑“D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop”目录下的hdfs-site.xml文件: <configuration>  <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.data.dir</name> <value>/D:/ProgramData/BigData/Hadoop/hadoop-2.7.4/workplace/data</value> </property> </configuration>

[4].编辑“D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop”目录下的yarn-site.xml文件: <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> </configuration>

[5].编辑“D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\etc\hadoop”目录下的hadoop-env.cmd文件，将JAVA_HOME用 @rem注释掉，编辑为JAVA_HOME的路径: set JAVA_HOME=D:/ProgramLanguage/Java/jdk/jdk1.8.0_144

[6].下载到的hadooponwindows-master.zip，解压，将bin目录(包含以下.dll和.exe文件)文件替换原来hadoop目录下的bin目录

[7].

hadoop fs -mkdir hdfs://localhost:8087/user hadoop fs -mkdir hdfs://localhost:8087/user/wcinput

hadoop fs -put D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\workplace\data\file1.txt hdfs://localhost:8087/user/wcinput hadoop fs -put D:\ProgramData\BigData\Hadoop\hadoop-2.7.4\workplace\data\file2.txt hdfs://localhost:8087/user/wcinput

hadoop fs -ls hdfs://localhost:8087/user/wcinput

转载于:https://my.oschina.net/2273990296/blog/1524434