- 关闭防火墙
- 查看防火墙状态 firewall-cmd –state
- 停止firewall systemctl stop firewalld.service
- 禁止firewall开机启动 systemctl disable firewalld.service
- .配置主机名
- 设置主机名 hostnamectl set-hostname hadoop01
- 查看主机名 hostname hostnamectl
- vim /etc/sysconfig/network
- 内容信息如下
- NETWORING=yes
- HOSTNAME=hadoop01
- 配置hosts文件
- 192.168.244.131 hadoop01
- 192.168.244.132 hadoop02
- 配置免密码登录
- 在hadoop01节点
- 执行ssh-keygen,一路回车即可
- 在 目录 ~/.ssh下生成 id_rsa.pub
- cat id_rsa.pub>> authorized_keys
- 在hadoop02节点执行相同操作
- 将hadoop01的id_rsa.pub内容追加在hadoop02的authorized_keys文件中
- 将hadoop02的id_rsa.pub内容追加在hadoop01的authorized_keys文件中、
- 将每个节点 chmod 700 ~/.ssh chmod 600 authorized_keys
- 每个节点 /bin/systemctl start sshd.service 启动ssh服务
- 重启系统 reboot
- 在hadoop01节点
- 安装jdk 参考linux目录下的安装 linux安装JDK
- 上传和解压hadoop安装包 tar -xvf hadoop-3.0.3-src.tar.gz
- 目录说明: bin目录:命令脚本 etc/hadoop:存放hadoop的配置文件 lib目录:hadoop运行的依赖jar包 sbin目录:启动和关闭hadoop等命令都在这里 libexec目录:存放的也是hadoop命令,但一般不常用
- 将hadoop的bin目录下的可执行文件加入到系统变量中
- vim /ect/profile
- #set java environment
- vim /ect/profile
JAVA_HOME=/usr/local/src/java/jdk1.8.0_151
HADOOP_HOME=/usr/local/src/hadoop/hadoop-3.0.3
JAVA_BIN=/usr/local/src/java/jdk1.8.0_151/bin
PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME JAVA_BIN PATH CLASSPATH HADOOP_HOME
-
- vim ~/.bash_profile
- # User specific environment and startup programs
- vim ~/.bash_profile
export JAVA_HOME=/usr/local/src/java/jdk1.8.0_151
export HADOOP_HOME=/usr/local/src/hadoop/hadoop-3.0.3
PATH=$JAVA_HOME/bin:$PATH:$HOME/bin:$HADOOP_HOME/bin
export PATH
source生效
有两种,不知道具体是那个生效。没有进行实验
Hadoop部分:(目录根据自己的路径)
- 进入到/usr/local/src/hadoop/hadoop-3.0.3/etc/hadoop
- 配置 hadoop-env.sh(vim hadoop-env.sh )
- export JAVA_HOME=/usr/local/src/java/jdk1.8.0_151
- export HADOOP_HOME=/usr/local/src/hadoop/hadoop-3.0.3
- export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
- source hadoop-env.sh 使得设置生效,使得hadoop命令可用
- 修改core-site.xml
- <configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop01:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/src/hadoop/tmp</value>
</property>
<property>
<name>io.sort.mb</name>
<value>1024</value>
</property>
</configuration>
- vim hdfs-site.xml
- <configuration>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/src/hadoop/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/src/hadoop/name</value>
</property>
</configuration>
- vim mapred-site.xml
- <configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.admin.user.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME</value>
</property>
</configuration>
- vim yarn-site.xml
- <configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop01</value>
</property>
<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
</configuration>
- 配置workers文件 (注意:3.0为workers文件 之前为slavels文件)
- hadoop01
- hadoop02
- /usr/local/src/hadoop/hadoop-3.0.3/sbin
- vim start-dfs.sh
- HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
- vim start-yarn.sh
- YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=root
YARN_NODEMANAGER_USER=root
- vim stop-yarn.sh
- YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=root
YARN_NODEMANAGER_USER=root
- vim stop-dfs.sh
- HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
以上文件添加在首行
21. 将hadoop目录scp到其他两台机器的相应目录下
scp /usr/local/src/hadoop/hadoop-3.0.3 hadoop02:/usr/local/src/hadoop/hadoop-3.0.3
scp /usr/local/src/hadoop/hadoop-3.0.3 hadoop03:/usr/local/src/hadoop/hadoop-3.0.3
22. cd /usr/local/src/hadoop/hadoop-3.0.3/sbin
start-all.sh
23. 访问 dfs集群状态