hadoop-2.7.1的安装

最新推荐文章于 2021-09-26 07:45:46 发布

豹变

最新推荐文章于 2021-09-26 07:45:46 发布

阅读量2k

点赞数 1

文章标签： MR hadoop

注意：官网提供的都是32位的安装包，64位的安装包需要自己编译

1 部署

192.168.100.200 master

192.168.100.201 slave1

192.168.100.202 slave2

2 三台安装jdk

[ root@localhost local]# ls

bin etc games hadoop-1.2.1.tar.gz include jdk-7u79-linux-x64.tar.gz lib lib64 libexec sbin share src

[ root@localhost local]# tar -zxvf jdk-7u79-linux-x64.tar.gz

[ root@localhost local]# vi /etc/profile

JAVA_HOME=/usr/local/jdk1.7.0_79

PATH=$JAVA_HOME/bin:$PATH

CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export JAVA_HOME

export PATH

export CLASSPATH

[ root@localhost local]# source /etc/profile

3 完全分布式安装部署

192.168.100.200 master

192.168.100.201 slave1

192.168.100.202 slave2

3.1 配置SSH（三台）

[ root@localhost local]# cd /root

[ root@localhost ~]# ssh-keygen -t rsa

按两次回车

[ root@localhost ~]# cd /root/.ssh

[ root@localhost .ssh]# cp id_rsa.pub authorized_keys //如果authorized_keys文件中存在的公钥，都可以免密码登陆

把各个节点的authorized_keys内容互相拷贝到对方文件中，使得彼此可以免密码登陆

[ root@localhost /]# vi /etc/hosts

192.168.100.200 master

192.168.100.201 slave1

192.168.100.202 slave2

192.168.100.200 localhost.localdomain --每一台节点这一行不一致

[ root@localhost /]# hostname

localhost.localdomain

3.2 安装 hadoop

[ root@localhost local]# tar -zxvf hadoop-2.7.1.tar.gz

[ root@localhost hadoop-2.7.1]# cd etc/hadoop/

配置hadoop-env.sh、 core-site.xml 、 hdfs-site.xml 、 mapred-site.xml 、yarn-site.xml 、slave

[ root@localhost conf]# vi hadoop-env.sh

export JAVA_HOME=/usr/local/jdk1.7.0_79

[ root@localhost conf]# vi core-site.xml //指定namenode的IP和端口

<name>fs.defaultFS</name>

</property>

</configuration>

[ root@localhost conf]# vi hdfs-site.xml //指定数据节点数据路径，指定数据块复制份数

<name>dfs.replication</name>

</property>

<name>dfs.permissions</name>

<value>false</value>

</property>

<name>dfs.namenode.secondary.http-address</name>

</property>

</configuration>

[ root@localhost hadoop-2.7.1]# cp etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml

[ root@localhost hadoop-2.7.1]# vi etc/hadoop/mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

<name>mapreduce.jobhistory.address</name>

</property>

<name>mapreduce.jobhistory.webapp.address</name>

</property>

</configuration>

[ root@localhost hadoop]# vi yarn-site.xml

                                <configuration>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<name>yarn.resourcemanager.address</name>

</property>

<name>yarn.resourcemanager.scheduler.address</name>

</property>

<name>yarn.resourcemanager.resource-tracker.address</name>

</property>

<name>yarn.resourcemanager.admin.address</name>

</property>

<name>yarn.resourcemanager.webapp.address</name>

</property>

</configuration>

[ root@localhost conf]# vi slaves //数据节点所在机器

192.168.100.201

192.168.100.202

3.3 向各个节点复制hadoop

[ root@localhost hadoop]# scp -r /usr/local/hadoop-2.7.1 192.168.100.201:/usr/local

[ root@localhost hadoop]# scp -r /usr/local/hadoop-2.7.1 192.168.100.202:/usr/local

3.4 格式化HDFS

在名称节点运行命令,即master

[ root@localhost hadoop-1.2.1]# cd /usr/local/hadoop-2.7.1/bin/

[ root@localhost bin]# ./hdfs namenode -format

出现successfully formatted 成功

在你重新格式化分布式文件系统之前，需要将文件系统中的数据先清除，否则，datanode将创建不成功。

3.5 启动集群

禁用防火墙

[ root@localhost bin]# systemctl stop firewalld.service

可以在名称节点上运行

[ root@localhost sbin]# ./start-all.sh

3.6 查看启动进程

[ root@localhost bin]# /usr/local/jdk1.7.0_79/bin/jps

master上

slave上

web访问

http://192.168.100.200:8088/ resourcemanager

http://192.168.100.200:50070/ namenode 可以看到live node 为2

4 eclipse集成hadoop开发环境

将 hadoop-eclipse-plugin-2.7.1.jar拷贝到${eclipse} \dropins\plugins

打开windows -->showview -->找到Map/Reduce Locaions -->new Hadoop location

将hadoop-2.7.1解压到本地一份，放d:\下

将对应的 winutils.exe和 hadoop.dll文件拷贝到hadoop/bin下面

之后将hadoop的路径配置为环境变量中

在项目的src下面新建log4j.properties

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender

log4j.appender.stdout.layout=org.apache.log4j.PatternLayout

log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - %m%n

log4j.appender.logfile=org.apache.log4j.FileAppender

log4j.appender.logfile.File=target/spring.log

log4j.appender.logfile.layout=org.apache.log4j.PatternLayout

log4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %m%n

豹变

关注

1
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
hadoop-2.7.1的安装

注意：官网提供的都是32位的安装包，64位的安装包需要自己编译 1 部署 192.168.100.200 master 192.168.100.201 slave1 192.168.100.202 slave22 三台安装jdk
复制链接

扫一扫