Hadoop完全分布式ha安装以及一些懒人设置
- Hadoop版本2.6.5
解压,修改配置文件,分发到另外3台,软件安排
Node | NN-1 | NN-2 | DN | ZK | ZKFC | JNN | Master | BM | RES |
---|---|---|---|---|---|---|---|---|---|
node01 | + | + | + | + | |||||
node02 | + | + | + | + | + | + | |||
node03 | + | + | + | + | |||||
node04 | + | + | + |
NN:NameNode
DN:DataNode
ZK:zookeeper
ZKFC:FailoverController
JNN:JournalNode
BM:BackupMaster
RES:Regionservers
2. JDK安装,免密钥设置
免密钥设置网上去找,必须达成免密钥。
配置环境变量:vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.7.0_67
export HADOOP_PREFIX=/opt/sxt/hadoop-2.6.5
export HIVE_HOME=/opt/sxt/apache-hive-1.2.1-bin
export HBASE_HOME=/home/hbase-0.98.12.1-hadoop2
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin:$HIVE_HOME/bin:$HBASE_HOME/bin
配置完成后 :source /etc/profile
3. node01配置文件
路径:/opt/sxt/hadoop-2.6.5/etc/hadoop
mapred-env.sh
hadoop-env.sh
yarn-env.sh
都修改为:
export JAVA_HOME=/usr/java/jdk1.7.0_67
core-site.xml文件修改:
hadoop.tmp.dir必须为空文件夹。
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluster</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/sxt/hadoop/ha</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>node02:2181,node03:2181,node04:2181</value>
</property>
</configuration>
hdfs-site.xml
这里面包含了一下zookeeper的相关设置。
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.nameservices</name>
<value>mycluster</value>
</property>
<property>
<name>dfs.ha.namenodes.mycluster</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn1</name>
<value>node01:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn2</name>
<value>node02:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.mycluster.nn1</name>
<value>node01:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.mycluster.nn2</name>
<value>node02:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://node01:8485;node02:8485;node03:8485/mycluster</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_dsa</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
</configuration>
slaves
node02
node03
node04