hadoop spark hive 集群配置（三）hadoop集群配置代码 centos7

hadoop

---

master

1. 修改 /etc/hosts 文件，操作过程如下图所示：

vi /etc/hosts

1. 10.117.0.180 master
2. 10.117.0.180 slave1
3. 10.117.0.180 slave2

scp -r /etc/hosts slave1:/etc/ 
scp -r /etc/hosts slave2:/etc/

测试是否成功

ssh slave1

解压hadoop,编辑profile

mkdir -p /usr/hadoop 
tar -zxvf /opt/soft/hadoop-2.7.3.tar.gz -C /usr/hadoop/ 
vim /etc/profile

1. export HADOOP_HOME=/usr/hadoop/hadoop-2.7.3
2. export CLASSPATH=$CLASSPATH:$HADOOP_HOME/lib
3. export PATH=$PATH:$HADOOP_HOME/bin

source /etc/profile

分发profile

scp -r /etc/profile slave1:/etc/ 
scp -r /etc/profile slave2:/etc/

slave1 slave 2

source /etc/profile

master

2.对hadoop进行配置，所有的hadoop的配置文件都在/usr/hadoop/hadoop-2.7.3/etc/hadoop

• #配置hadoop-env

cd /usr/hadoop/hadoop-2.7.3/etc/hadoop 
vim hadoop-env.sh

1. export JAVA_HOME=/usr/java/jdk1.8.0_171

• #配置core-site.xml 
  vim core-site.xml

1. <configuration>
2. <property>
3. <name>fs.default.name</name>
4. <value>hdfs://master:9000</value>
5. </property>
6. <property>
7. <name>hadoop.tmp.dir</name>
8. <value>/usr/hadoop/hadoop-2.7.3/hdfs/tmp</value>
9. <description>A base for other temporary directories.</description>
10. </property>
11. <property>
12. <name>io.file.buffer.size</name>
13. <value>131072</value>
14. </property>
15. <property>
16. <name>fs.checkpoint.period</name>
17. <value>60</value>
18. </property>
19. <property>
20. <name>fs.checkpoint.size</name>
21. <value>67108864</value>
22. </property>
23. </configuration>

• #使用vim命令编辑yarn-site.xml文件 
  vim yarn-site.xml

1. <configuration>
2. <property>
3. <name>yarn.resourcemanager.address</name>
4. <value>master:18040</value>
5. </property>
6. <property>
7. <name>yarn.resourcemanager.scheduler.address</name>
8. <value>master:18030</value>
9. </property>
10. <property>
11. <name>yarn.resourcemanager.webapp.address</name>
12. <value>master:18088</value>
13. </property>
14. <property>
15. <name>yarn.resourcemanager.resource-tracker.address</name>
16. <value>master:18025</value>
17. </property>
18. <property>
19. <name>yarn.resourcemanager.admin.address</name>
20. <value>master:18141</value>
21. </property>
22. <property>
23. <name>yarn.nodemanager.aux-services</name>
24. <value>mapreduce_shuffle</value>
25. </property>
26. <property>
27. <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
28. <value>org.apache.hadoop.mapred.ShuffleHandler</value>
29. </property>
30. <!-- Site specific YARN configuration properties -->
31. </configuration>

• #编辑slaves文件 
  vim slaves

1. slave1
2. slave2

• #编辑master文件 
  vim master

1. master

• #编辑hdfs文件 
  vim hdfs-site.xml

1. <configuration>
2. <property>
3. <name>dfs.replication</name>
4. <value>2</value>
5. </property>
6. <property>
7. <name>dfs.namenode.name.dir</name>
8. <value>file:/usr/hadoop/hadoop-2.7.3/hdfs/name</value>
9. <final>true</final>
10. </property>
11. <property>
12. <name>dfs.datanode.data.dir</name>
13. <value>file:/usr/hadoop/hadoop-2.7.3/hdfs/data</value>
14. <final>true</final>
15. </property>
16. <property>
17. <name>dfs.namenode.secondary.http-address</name>
18. <value>master:9001</value>
19. </property>
20. <property>
21. <name>dfs.webhdfs.enabled</name>
22. <value>true</value>
23. </property>
24. <property>
25. <name>dfs.permissions</name>
26. <value>false</value>
27. </property>
28. </configuration>

• #使用vim命令编辑mapred-site.xmll 
  cp mapred-site.xml.template mapred-site.xml 
  vim mapred-site.xml

1. <property>
2. <name>mapreduce.framework.name</name>
3. <value>yarn</value>
4. </property>

分发软件和软件配置文件

scp -r /usr/hadoop root@slave1:/usr/ 
scp -r /usr/hadoop root@slave2:/usr/

格式化

hadoop namenode -format

开启

cd /usr/hadoop/hadoop-2.7.3 
sbin/start-all.sh

master,slave1 slave2

查看主进程

jps

输入网址50070访问

在浏览器输入 masterIP：50070

后续shell查看

hadoop fs -ls / 
hadoop fs -mkdir /data 
hadoop fs -ls /

“Utilities”->“Browse the file system”