1.hadoop集群搭建

最新推荐文章于 2022-05-09 22:11:43 发布

bcsix

最新推荐文章于 2022-05-09 22:11:43 发布

阅读量534

点赞数

分类专栏： hadoop 文章标签： hadoop hadoop集群

本文链接：https://blog.csdn.net/bcsix/article/details/52872537

版权

hadoop 专栏收录该内容

8 篇文章 0 订阅

订阅专栏

本文适合对hadoop和linux稍有了解的人群，新手请观看其他更为详细的博文

集群规划：

主机名	IP	安装的软件	运行的进程	说明
hadoop001	192.168.100.101	jdk、hadoop	NameNode、DFSZKFailoverController
hadoop002	192.168.100.102	jdk、hadoop	NameNode、DFSZKFailoverController
hadoop003	192.168.100.103	jdk、hadoop	ResourceManager
hadoop004	192.168.100.104	jdk、hadoop	ResourceManager
hadoop005	192.168.100.105	jdk、hadoop、zookeeper	DataNode、NodeManager、JournalNode、QuorumPeerMain
hadoop006	192.168.100.106	jdk、hadoop、zookeeper	DataNode、NodeManager、JournalNode、QuorumPeerMain
hadoop007	192.168.100.107	jdk、hadoop、zookeeper	DataNode、NodeManager、JournalNode、QuorumPeerMain

1.基本环境配置

1.1修改主机名

vim /etc/sysconfig/network(集群的每一个服务器都要修改，给一个起个正规的名字)

NETWORKING=yes

HOSTNAME= hadoop001

1.2修改IP

vim /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE="eth0"

BOOTPROTO=" static"

HWADDR="00:0C:29:3C:BF:E7"

IPV6INIT="yes"

NM_CONTROLLED="yes"

ONBOOT="yes"

TYPE="Ethernet"

UUID="ce22eeca-ecde-4536-8cc2-ef0dc36d4a8c"

IPADDR="192.168.100.101 "

NETMASK="255.255.255.0"

GATEWAY="192.168.100 .1"

1.3修改主机名和IP的映射关系

vim /etc/hosts

192.168.100.101 hadoop001

192.168.100.102 hadoop002

192.168.100.103 hadoop003

192.168.100.104 hadoop004

192.168.100.105 hadoop005

192.168.100.106 hadoop006

scp /etc/hosts root@hadoop2:/etc/hosts

1.4关闭防火墙

chkconfig iptables off

1.5重启Linux ,测试网络配置和防火墙配置

reboot

//新建用户

groupadd hadoop

useradd -g hadoop hadoop

passwd hadoop

//使用hadoop用户登录

1.6生成ssh密匙

ssh-keygen -t rsa

//拷贝到相应主机(按照hadoop规划拷贝)

ssh-copy-id -i hadoop001

1.7安装jdk

tar -zxvf jdk.xxx.tar.gz -C /opt/

vim /etc/profile

export JAVA_HOME=/opt/jdk.xxx

export PATH=$PATH:$JAVA_HOME/bin

source /etc/profile

2安装Hadoop

2.1解压

tar -zxvf hadoop-xxx.tar.gz -C /opt/

2.2添加环境变量

vim /etc/profile

export JAVA_HOME=/opt/jdk.xxx

export HADOOP_HOME=/opt/hadoop-xxx

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin

2.3开始hadoop设置

cd /opt/hadoop-xxx/etc/hadoop

1.vim hadoo-env.sh

export JAVA_HOME=/opt/jdk.xxx

2.vim core-site.xml

<name>fs.defaultFS</name>

<value>hdfs://nameservice1</value>

</property>

<name>hadoop.tmp.dir</name>

<value>/opt/hadoop-xxx/tmp</value>

</property>

<name>ha.zookeeper.quorum</name>

<value>hadoop005,hadoop006,hadoop007</value>

</property>

</configuration>

3.vim hdfs-site.xml

<name>dfs.nameservices</name>

<value>nameservice 1</value>

</property>

<name>dfs.ha.namenodes.nameservice1</name>

<value>namenode 1,namenode 2</value>

</property>

<name>dfs.namenode.rpc-address.nameservice 1.namenode1</name>

<value>hadoop001:9000</value>

</property>

<name>dfs.namenode.http-address.nameservice1.namenode 1</name>

<value>hadoop001:50070</value>

</property>

<name>dfs.namenode.rpc-address.nameservice1.namenode 2</name>

<value>hadoop002:9000</value>

</property>

<name>dfs.namenode.http-address.nameservice1.namenode 2</name>

<value>hadoop002:50070</value>

</property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://hadoop005:8485;hadoop006:8485;hadoop007:8485/ nameservice1</value>

</property>

<name>dfs.journalnode.edits.dir</name>

<value>/opt/hadoop-xxx/journal</value>

</property>

<name>dfs.ha.automatic-failover.enabled</name>

</property>

<name>dfs.client.failover.proxy.provider. nameservice1</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<name>dfs.ha.fencing.methods</name>

<value>

sshfence

shell(/bin/true)

</value>

</property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/home/hadoop/.ssh/id_rsa</value>

</property>

<name>dfs.ha.fencing.ssh.connect-timeout</name>

</property>

</configuration>

4.vim mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

</configuration>

5.vim yarn-site.xml

<name>yarn.resourcemanager.ha.enabled</name>

</property>

<name>yarn.resourcemanager.cluster-id</name>

</property>

<name>yarn.resourcemanager.ha.rm-ids</name>

</property>

<name>yarn.resourcemanager.hostname.rm1</name>

<value>hadoop003</value>

</property>

<name>yarn.resourcemanager.hostname.rm2</name>

<value>hadoop004</value>

</property>

<name>yarn.resourcemanager.zk-address</name>

<value>hadoop005:2181,hadoop006:2181,hadoop007:2181</value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

6.vim slaves(指定小弟，记住hadoop001,002和hadoop003,004如果小弟不同需要分别指定不同的小弟)

hadoop005

hadoop006

hadoop007

7.将配置好的hadoop拷贝到其他节点

scp -r /opt/hadoop-xxx hadoop@hadoopxxx :/opt/

2.4安装配置zooekeeper

tar -zxvf zookeeper-xxx.tar.gz -C /opt/

cd /opt/zookeeper-xxx/conf/

cp zoo_sample.cfg zoo.cfg

vim zoo.cfg

修改：dataDir=/opt/zookeeper-xxx/tmp

在最后添加：

server.001=hadoop05:2888:3888

server.002=hadoop06:2888:3888

server.003=hadoop07:2888:3888

然后创建一个tmp文件夹

mkdir /opt/zookeeper-xxx/tmp

再创建一个空文件

touch /opt/zookeeper-xxx/tmp/myid

最后向该文件写入ID

echo 001 > /itcast/zookeeper-xxx/tmp/myid

拷贝到其他要安装zookeeper的节点

scp -r /opt/zookeeper-xxx/ hadoop@hadoopxxx:/opt/

修改myid为对应的id

2.5分别启动zooekeeper

cd /opt/zookeeper-xxx/bin/

./zkServer.sh start

./zkServer.sh status

jps检测

2.6启动journalnode（在hadoop001上完成）

cd /itcast/hadoop-xxx/sbin

./hadoop-daemons.sh start journalnode

jps检测

2.7格式化HDFS

hdfs namenode -format

2.8将生产的数据文件夹拷贝到其他namenode

scp -r /opt/hadoop-xxx/tmp/ hadoop@hadoopxxx:/opt/hadoop-xxx/

2.9格式化ZK(在hadoop001上执行即可)

hdfs zkfc -formatZK

2.10启动HDFS(在hadoop001上执行)

sbin/start-dfs.sh

2.11启动YARN(在hadoop003上执行)

sbin/start-yarn.sh

最后使用jps在各节点上检测进程是否启动，如果按照规划的都启动了一般就没问题了

可通过http://hadoop001的地址:50070就可以看到web管理界面

最后说说我遇到的坑，

主机名的命名：我先前为了容易区分将主机名命名为CentOS_bcsix_001的形式，结果在HDFS格式化的时候就报错了，最后换成了hadoop001(去掉下划线等特殊符号)这种形式就对了

本文参考吴超老师“hadoop7天培训”，只为熟悉hadoop集群搭建过程，有错误之处，望大家批评指正。

bcsix

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
1.hadoop集群搭建

本文适合对hadoop和linux稍有了解的人群，新手请观看其他更为详细的博文主机名IP安装的软件运行的进程说明hadoop001192.168.100.101jdk、hadoopNameNode、DFSZKFailoverController had
复制链接

扫一扫