Hadoop cluster HA 客户端操作 Yarn高可用

最新推荐文章于 2024-04-22 10:01:40 发布

qingwei201314

最新推荐文章于 2024-04-22 10:01:40 发布

阅读量264

点赞数

本文链接：https://blog.csdn.net/qingwei201314/article/details/117703649

版权

一、通过QJM模式部署Hadoop HA

三台机器：192.168.86.130、192.168.86.131、192.168.86.132

1.部署zookeeper，分别在三台机器上部署zookeeper, 配置文件zoo.cfg内容如下：

tickTime=2000

dataDir=/usr/local/zookeeper-3.6.1/data

clientPort=2181

initLimit=5

syncLimit=2

server.130=192.168.86.130:2888:3888

server.131=192.168.86.131:2888:3888

server.132=192.168.86.132:2888:3888

2.三台zookeeper配置一样，配置完成后，分别启动：./zkServer.sh start

3.部署Hadoop：

namenode的机器共两台：192.168.86.130、192.168.86.131.配置内容下如：

3.1core-site.xml内容如下：

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>fs.defaultFS</name>

<value>hdfs://hadoop-cluster</value>

</property>

<name>hadoop.tmp.dir</name>

<value>/home/hadoop_data</value>

</property>

<name>hadoop.proxyuser.root.hosts</name>

</property>

<name>hadoop.proxyuser.root.groups</name>

</property>

<name>ha.zookeeper.quorum</name>

</property>

<name>ha.zookeeper.auth</name>

<value>@/usr/local/hadoop-2.10.0/security/zk-auth.txt</value>

</property>

<name>ha.zookeeper.acl</name>

<value>@/usr/local/hadoop-2.10.0/security/zk-acl.txt</value>

</property>

</configuration>

3.2 hdfs-site.xml内容如下：

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>dfs.replication</name>

</property>

<name>dfs.datanode.data.dir</name>

<value>/home/hadoop_data</value>

</property>

<name>dfs.hosts</name>

<value>/usr/local/hadoop-2.10.0/etc/hadoop/datanode_list.txt</value>

</property>

<name>dfs.nameservices</name>

<value>hadoop-cluster</value>

</property>

<name>dfs.ha.namenodes.hadoop-cluster</name>

</property>

<name>dfs.namenode.rpc-address.hadoop-cluster.nn1</name>

<value>hadoop-130:8020</value>

</property>

<name>dfs.namenode.rpc-address.hadoop-cluster.nn2</name>

<value>hadoop-131:8020</value>

</property>

<name>dfs.namenode.http-address.hadoop-cluster.nn1</name>

</property>

<name>dfs.namenode.http-address.hadoop-cluster.nn2</name>

</property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://192.168.86.130:8485;192.168.86.131:8485/hadoop-cluster</value>

</property>

<name>dfs.client.failover.proxy.provider.hadoop-cluster</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

</property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

</property>

<name>dfs.journalnode.edits.dir</name>

<value>/home/hadoop_data/journalnode</value>

</property>

<name>dfs.namenode.name.dir</name>

<value>/home/hadoop_data/namenode</value>

</property>

<name>dfs.ha.automatic-failover.enabled</name>

</property>

</configuration>

4.192.168.86.132只作为datanode，配置文件如下：
4.1 core-site.xml内容如下：

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>hadoop.tmp.dir</name>

<value>/home/hadoop_data</value>

</property>

<name>hadoop.proxyuser.root.hosts</name>

</property>

<name>hadoop.proxyuser.root.groups</name>

</property>

</configuration>

4.2 hdfs-site.xml内容如下：

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>dfs.replication</name>

</property>

<name>dfs.datanode.data.dir</name>

<value>/home/hadoop_data</value>

</property>

<name>dfs.nameservices</name>

<value>hadoop-cluster</value>

</property>

<name>dfs.ha.namenodes.hadoop-cluster</name>

</property>

<name>dfs.namenode.rpc-address.hadoop-cluster.nn1</name>

<value>hadoop-130:8020</value>

</property>

<name>dfs.namenode.rpc-address.hadoop-cluster.nn2</name>

<value>hadoop-131:8020</value>

</property>

</configuration>

5.datanode_list.txt的内容如下：

hadoop-130

hadoop-131

hadoop-132

6. 在三台机器上配置hosts, vi /etc/hosts

192.168.86.131 hadoop-131

192.168.86.130 hadoop-130

192.168.86.132 hadoop-132

6. 在三台机器上分别启动journalnode: nohup ./hdfs journalnode >journalnode.log 2>&1 &

7.在192.168.86.130或192.168.86.131上启动namenode集群：sbin/start-dfs.sh

8.在三台机器上分别启动datanode: nohup ./hdfs datanode>datanode.log 2>&1 &

附录：
1.ssh的免密登录配置详见操作手册，生成id_rsa的方法需要百度搜一下具体的操作步骤。
2.zookeeper的安全登录配置详见操作手册。

二、Java客户端操作Hadoop代码如下：

Configuration conf = new Configuration();
conf.set("dfs.replication","3");
conf.set("dfs.nameservices", "hadoop-cluster");
conf.set("dfs.ha.namenodes.hadoop-cluster", "nn1,nn2");
conf.set("dfs.namenode.rpc-address.hadoop-cluster.nn1", "hadoop-130:8020");
conf.set("dfs.namenode.rpc-address.hadoop-cluster.nn2", "hadoop-131:8020");
conf.set("dfs.client.failover.proxy.provider.hadoop-cluster", "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");
FileSystem fs =FileSystem.get(new URI("hdfs://hadoop-cluster"), conf, "root");
fs.mkdirs(new Path("/kevin/canal/"));
FSDataOutputStream out = null;
out = fs.create(new Path("/kevin/canal/canal-0330-2.txt"));
for(int j = 0; j < 10000; j ++){
    String name = "Kevin" + j;
out.writeBytes(j + "," + name + "," + new Date());
out.writeBytes("\n");
System.out.println(j);
}
out.flush();
out.close();
fs.close();

客户端的host需要增加配置：

192.168.86.130 hadoop-130

192.168.86.131 hadoop-131

三、Yarn高可用
1.分别在192.168.86.130和192.168.86.131两台机器上修改yarn的配置文件，两台机器配置相同，yarn-site.xml配置文件内容如下：

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.nodemanager.env-whitelist</name>

<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>

</property>

<name>yarn.resourcemanager.recovery.enabled</name>

</property>