安装 hadoop
赋予hadoop用户sudo权限:
1. 切换到root用户,给hadoop用户授予sudo权限:修改/etc/sudoers文件,由于默认root用户也不可以写入,这里开启写入权限:
su - root
cd /etc/
ls -la | grep sudoer
chmod u+w sudoers
2 增加一行,赋予hadoop用户sudo权限: hadoop ALL=(ALL) ALL
3.用root用户在/usr/local目录下面解压hadoop压缩包
cd /soft
ls -lh
tar -zxvf hadoop-2.7.4.tar.gz
mv hadoop-2.7.4 /usr/local/hadoop-2.7.4
cd /usr/local/hadoop-2.7.4
mkdir tmp
mkdir dfs
改变文件的用户组:
chown -R hadoop /usr/local/hadoop-2.7.4/
chgrp -R hadoop /usr/local/hadoop-2.7.4/
在/etc/profile 中配置hadoop的环境变量:
echo "export HADOOP_HOME=/usr/local/hadoop-2.7.4" >> /etc/profile
echo -e 'export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin'>> /etc/profile
source /etc/profile
控制台 hadoop version 查看hadoop版本信息: hadoop version
修改hadoop-env.sh:
vim打开:
vim /usr/local/hadoop-2.7.4/etc/hadoop/hadoop-env.sh
修改如下行:
export JAVA_HOME=/usr/java/jdk1.8.0
修改:vim /usr/local/hadoop-2.7.4/etc/hadoop/core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-2.7.4/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://node1:9000</value>
</property>
</configuration>
su - hadoop
cd ~
mkdir tmp
修改:vim /usr/local/hadoop-2.7.4/etc/hadoop/hdfs-site.xml
configuration>
<property>
<name>dfs.name.dir</name>
<value>/usr/local/hadoop-2.7.4/dfs/name</value>
<description>Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently.</description>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/local/hadoop-2.7.4/dfs/data</value>
<description>Comma separated list of paths on the local filesystem of a DataNode where it should store its blocks.</description>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
修改:vim /usr/local/hadoop-2.7.4/etc/hadoop/yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>node1</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>node1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>node1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>node1:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>node1:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>node1:8088</value>
</property>
</configuration>
修改:/usr/local/hadoop-2.7.4/etc/hadoop/mapred-site.xml
mv mapred-site.xml.template mapred-site.xml
vim /usr/local/hadoop-2.7.4/etc/hadoop/mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.job.tracker</name>
<value>node1:9001</value>
</property>
<property>
<name>mapreduce.jobtracker.http.address</name>
<value>node1:50030</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>node1:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>node1:19888</value>
</property>
配置masters文件:
cd /usr/local/hadoop-2.7.4/etc/hadoop/
vim masters
添加:
node1
vim slaves
添加并注释原文件中的localhost:
#localhost
node2
node3
格式户hdfs:
hadoop namenode -format
启动:
/usr/local/hadoop-2.7.4/sbin/./start-all.sh
更推荐的运行方式:
cd sbin
./start-dfs.sh
./start-yarn.sh
单独启动datanode或者namenode
hadoop-daemon.sh start datanode
hadoop-daemon.sh start namenode
停止:
/usr/local/hadoop-2.7.4/sbin/./stop-all.sh
访问查看启动状态可以访问:
http://192.168.209.129:50070/
http://192.168.209.129:8088
jps查看:
主节点: NameNode, SecondNameNode, ResourceManager已经启动
从节点:NodeManager,dataNode已经启动
主节点监控运行状态:
hadoop dfsadmin -report
查看端口占用:
netstat -ntlp
netstat -tnl
centos7 hadoop 3节点集群安装笔记
最新推荐文章于 2023-11-01 23:46:51 发布