centos7 安装hadoop2.6.1 集群
1、规划:
ip hostname 角色
192.168.157.60 master master
192.168.157.61 slave1 salve
192.168.157.62 salve2 salve
2、虚拟机安装
2.1 安装 centos 7.5 虚拟机 ,选择最小安装。
2.2 安装网络工具 yum install net-tools
2.3 安装 vim yum -y install vim
3、修改为静态ip
vi /etc/sysconfig/network-scripts/ifcfg-ens33
修改 :
BOOTPROTO=static
ONBOOT=yes
添加:
BOOTPROTO=static
IPADDR=192.168.157.60
NETMASK=255.255.255.0
GATEWAY=192.168.157.2
DNS1=180.76.76.76
4、修改主机名
master机器: hostname master
修改 /etc/sysconfig/network
添加:
NETWORKING=yes
HOSTNAME=master
slave待会要做同样的修改
5、修改hosts文件
vi /etc/hosts
添加:
192.168.157.60 master
192.168.157.61 slave1
192.168.157.62 slave2
slave机器也一样修改
6、关闭防-火-墙
systemctl stop firewalld
systemctl disable firewalld
查看:firewall-cmd --state
7、配置ssh免密码登录
ssh-keygen
cd ~/.ssh
touch authorized_keys
cat id_rsa.pub > authorized_keys
在slave机器上 也执行 ssh-keygegn
把公钥添加到 master的 authorized_keys
然后 把 authorized_keys 传到另另个机器上
8、安装jdk 1.8 配置HADOOP_HOME
vi ~/.bashrc 添加:
export JAVA_HOME=/usr/local/src/jdk1.8.0_172
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export HADOOP_HOME=/usr/local/src/hadoop-2.6.1
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin
使生效: source ~/.bashrc
测试 : java -version
9、配置 hadoop 2.6.1
9.0 修改slaves
vi slaves:
slave1
slave2
9.1 修改 hadoop-env.sh
export JAVA_HOME=/usr/local/src/jdk1.8.0_172
9.2 修改 core-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/src/hadoop-2.6.1/tmp</value> </property> <property> <name>fs.default.name</name> <value>hdfs://192.168.157.60:9000</value> </property> </configuration>
要创建 /usr/local/src/hadoop-2.6.1/tmp
9.3 修改 hdfs-site.xml
<property> <name>dfs.name.dir</name> <value>/usr/local/src/hadoop-2.6.1/dfs/name</value> </property> <property> <name>dfs.data.dir</name> <value>/usr/local/src/hadoop-2.6.1/dfs/data</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property>
9.4 修改 mapred-site.xml
<property> <name>mapreduce.framework.name</name> <value>yarn</value> </property>
9.5 修改 yarn-site.xml
<property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>master:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>master:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>master:8035</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>master:8033</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>master:8088</value> </property>
10 、 启动集群
格式化 hdfs ./hdfs namenode -format
sbin/start-all.sh
错误是:The authenticity of host xxx.xxx.xxx.xxx can't be established。
执行ssh -o StrictHostKeyChecking=no yyy.yyy.yyy.yyy,就可以了。yyy.yyy.yyy.yyy是本地机器的IP地址
[root@master sbin]# jps
27552 Jps
27284 ResourceManager
27144 SecondaryNameNode
26959 NameNode