hadoop是java开发的,其运行的环境必须是java环境,在vmvare中安装CentOS7并配置网络和jdk环境中已配置好,这里需要四台机器作为集群配置
四台的主机名和IP地址及安装的服务如下表:
每台都要设置本机到主机的映射
vi /etc/hosts
在问答末尾添加
关闭防火墙
systemctl stop firewalld.service
systemctl disable firewalld.service
关闭selinux
vi /etc/selinux/config
SELINUX=disabled
每台都要配置jdk和Hadoop的环境变量
vi /etc/profile
source /etc/profile (或 . /etc/profile 文件加载一下)
ssh免密设置
ssh免密是为了启动start-dfs.sh,start-dfs.sh在哪里启动就要对别人公开自己的公钥,每台机子都执行
ssh localhost
执行ssh localhost有两个作用,1、验证自己还没免密 2、被动生成了 /root/.ssh目录(这个目录最好不要自己创建,涉及到权限配置,最好自动生成)
以node01登录任何机器都可以免密,所有将自己的公钥分发给node02,node03,node04
scp /root/.ssh/id_dsa.pub node02:/root/.ssh/node01.pub
scp /root/.ssh/id_dsa.pub node03:/root/.ssh/node01.pub
scp /root/.ssh/id_dsa.pub node04:/root/.ssh/node01.pub
然后在node02,node03,node04分别执行如下命令
cd ~/.ssh
cat node01.pub >> authorized_keys
给出NameNode角色在哪里启动
cd $HADOOP_HOME/etc/hadoop
vi core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://node01:9000</value>
</property>
</configuration>
vi hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/var/bigdata/hadoop/full/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/var/bigdata/hadoop/full/dfs/data</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>node02:50090</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/var/bigdata/hadoop/full/dfs/secondary</value>
</property>
</configuration>
配置DN这个角色再哪里启动
vi slaves
node02
node03
node04
分发:
cd /opt
scp -r ./bigdata/ node02:`pwd`
scp -r ./bigdata/ node03:`pwd`
scp -r ./bigdata/ node04:`pwd`
格式化启动
hdfs namenode -format
start-dfs.sh
上面完整的集群完毕,下面是从上面的搭建的基础上进行HA(高可用)的搭建,真正实现下表
在HA模式下,有两处需要用到免密:
1)启动start-dfs.sh脚本的机器需要将公钥分发别的节点(这个在上面已经设置好了)
2)在HA模式下,每一个NN身边会启动ZKFC,ZKFC会用免密的方式控制自己和其他NN节点的NN状态
在node02创建公钥
cd ~/.ssh
ssh-keygen -t dsa -P '' -f ./id_dsa
cat id_dsa.pub >> authorized_keys
scp ./id_dsa.pub node01:`pwd`/node02.pub
在node01上操作
cd ~/.ssh
cat node02.pub >> authorized_keys
这样node01 和 node02可以互相免密登录
在node02,node03,node04部署zookeeper
安装在/opt/bigdata目录里面(apache-zookeeper-3.6.2)
cd /opt/bigdata/apache-zookeeper-3.6.2/conf
cp zoo_sample.cfg zoo.cfg
vi zoo.cfg
cd /var/bigdata/hadoop'
mkdir zk
echo 1 > myid
vi /etc/profile
scp -r ./apache-zookeeper-3.6.2 node03:`pwd`
scp -r ./apache-zookeeper-3.6.2 node04:`pwd`
在node03上操作
mkdir /var/bigdata/hadoop/zk
echo 2 > /var/bigdata/hadoop/zk/myid
vi /etc/profile
在node04上操作
mkdir /var/bigdata/hadoop/zk
echo 3 > /var/bigdata/hadoop/zk/myid
vi /etc/profile
node02,node03,node04
zkServer.sh start
备注:有可能启动失败,是因为apache-zookeeper-3.6.2中缺少一个lib文件夹,打开https://mirrors.bfsu.edu.cn/apache/zookeeper/zookeeper-3.6.2/
下载第一个,然后解压,将里面的lib文件夹上传到apache-zookeeper-3.6.2目录下即可,或者你可以试着直接用第一个压缩安装zookeeper试一下是否可以
zkServer.sh status,可以查看zookeeper的状态
修改node01配置文件
cd /opt/bigdata/hadoop-2.10.1/etc/hadoop
vi core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluster</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>node02:2181,node03:2181,node04:2181</value>
</property>
</configuration>
vi hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/var/bigdata/hadoop/ha/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/var/bigdata/hadoop/ha/dfs/data</value>
</property>
<property>
<name>dfs.nameservices</name>
<value>mycluster</value>
</property>
<property>
<name>dfs.ha.namenodes.mycluster</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn1</name>
<value>node01:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn2</name>
<value>node02:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.mycluster.nn1</name>
<value>node01:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.mycluster.nn2</name>
<value>node02:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://node01:8485;node02:8485;node03:8485/mycluster</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/var/bigdata/hadoop/ha/dfs/jn</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_dsa</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
</configuration>
分发配置
scp core-site.xml hdfs-site.xml node02:`pwd`
scp core-site.xml hdfs-site.xml node03:`pwd`
scp core-site.xml hdfs-site.xml node04:`pwd`
初始化
1)先启动JN hadoop-daemon.sh start journalnode
2)选择一个NN做格式化(node01):hdfs namenode -format
3)启动这个格式化的NN(node01):以备另外一台同步 hadoop-daemon.sh start namenode
4)在另外一台机器中(node02):hdfs namenode -bootstrapStandby
5)格式化zookeeper(node01): hdfs zkfc -formatZK
6) 启动(node01):start-dfs.sh
IDEA使用Hadoop
使用的JDK要和Hadoop版本一致
pom.xml引入响应的包
将目录下/opt/bigdata/hadoop-2.10.1/etc/hadoop的core-site.xml和hdfs-site.xml文件拷贝到工程中
IDEA使用Hadoop的配置完成,使用代码如下:
package com.mingtimes.hdfs;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.io.IOUtils;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
/**
* @Description: com.mingtimes.hdfs
* @Date: 2021/4/6 19:00
*/
public class TestHdfs {
public Configuration conf = null;
public FileSystem fs = null;
@Before
public void conn() throws IOException {
System.setProperty("HADOOP_USER_NAME","god");
conf = new Configuration(true);
fs = FileSystem.get(conf);
// <property>
// <name>fs.defaultFS</name>
// <value>hdfs://mycluster</value>
// </property>
}
@Test
public void mkdir() throws IOException {
Path dir = new Path("/mingtimes");
if (fs.exists(dir)) {
fs.delete(dir,true);
}
fs.mkdirs(dir);
}
@Test
public void upload() throws Exception {
BufferedInputStream input = new BufferedInputStream(new FileInputStream(new File("./data/hello.txt")));
Path outfile = new Path("/mingtimes/out.txt");
FSDataOutputStream output = fs.create(outfile);
IOUtils.copyBytes(input,output,conf,true);
}
@Test
public void blocks() throws IOException {
Path file = new Path("/user/god/data.txt");
FileStatus fss = fs.getFileStatus(file);
BlockLocation[] blks = fs.getFileBlockLocations(fss, 0, fss.getLen());
for (BlockLocation blk : blks) {
System.out.println(blk);
}
//计算向数据移动!
//其实用户和程序读取的是文件这个级别,并不知道有块的概念
FSDataInputStream in = fs.open(file);//面向文件打开的输入流 无论怎么读都是从文件开始读起!
in.seek(1048576);
//计算向数据移动后,期望的是分治,只读自己关系(通过seek实现),同时,具备距离的概念(优先和本地的DN获取数据---框架的默认机制)
System.out.println((char)in.readByte());
System.out.println((char)in.readByte());
System.out.println((char)in.readByte());
System.out.println((char)in.readByte());
System.out.println((char)in.readByte());
System.out.println((char)in.readByte());
System.out.println((char)in.readByte());
System.out.println((char)in.readByte());
System.out.println((char)in.readByte());
System.out.println((char)in.readByte());
System.out.println((char)in.readByte());
System.out.println((char)in.readByte());
}
@After
public void close() throws IOException {
fs.close();
}
}