linux安装hadoop图文教程,Ubuntu 16.04安装 Hadoop 2.8.3 图文教程

环境：Ubuntu 16.04

需要软件：jdk ssh

https://mirrors.tuna.tsinghua.edu.cn/apache/Hadoop/common/

Hadoop 2.8.3

安装 jdk并配置环境变量

安装ssh和rshync，主要设置免密登录

sudo apt-get install ssh

sudo apt-get install rshync

sh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

ssh

安装hadoop

root@linuxidc.com:/usr/local/hadoop# tar -xzvf /home/hett/Downloads/hadoop-2.8.3.tar.gz

root@linuxidc.com:/usr/local/hadoop# mv hadoop-2.8.3  hadoop

root@linuxidc.com:/usr/local# cd hadoop/

root@linuxidc.com:/usr/local/hadoop# mkdir tmp

root@linuxidc.com:/usr/local/hadoop# mkdir hdfs

root@linuxidc.com:/usr/local/hadoop# mkdir hdfs/data

root@linuxidc.com:/usr/local/hadoop# mkdir hdfs/name

root@linuxidc.com:/usr/local/hadoop# nano /etc/profile

配置

export HADOOP_HOME=/usr/local/hadoop

export JAVA_HOME=/usr/local/jdk1.8.0_151

export JRE_HOME=${JAVA_HOME}/jre

export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib

export PATH=${JAVA_HOME}/bin:$PATH:$HADOOP_HOME/bin

root@linuxidc.com:/usr/local/hadoop# source /etc/profile

root@linuxidc.com:/usr/local/hadoop# cd etc/hadoop/

root@linuxidc.com:/usr/local/hadoop/etc/hadoop# ls

capacity-scheduler.xml      httpfs-env.sh            mapred-env.sh

configuration.xsl          httpfs-log4j.properties  mapred-queues.xml.template

container-executor.cfg      httpfs-signature.secret  mapred-site.xml.template

core-site.xml              httpfs-site.xml          slaves

hadoop-env.cmd              kms-acls.xml            ssl-client.xml.example

hadoop-env.sh              kms-env.sh              ssl-server.xml.example

hadoop-metrics2.properties  kms-log4j.properties    yarn-env.cmd

hadoop-metrics.properties  kms-site.xml            yarn-env.sh

hadoop-policy.xml          log4j.properties        yarn-site.xml

hdfs-site.xml              mapred-env.cmd

root@linuxidc.com:/usr/local/hadoop/etc/hadoop#

root@linuxidc.com:/usr/local/hadoop/etc/hadoop# nano hadoop-env.sh

export  JAVA_HOME=/usr/local/jdk1.8.0_151

配置yarn-env.sh

export  JAVA_HOME=/usr/local/jdk1.8.0_151

3)配置core-site.xml

添加如下配置：

fs.default.name

hdfs://localhost:9000

HDFS的URI，文件系统://namenode标识:端口号

hadoop.tmp.dir

/usr/hadoop/tmp

namenode上本地的hadoop临时文件夹

4)，配置hdfs-site.xml

添加如下配置

dfs.name.dir

/usr/hadoop/hdfs/name

namenode上存储hdfs名字空间元数据

dfs.data.dir

/usr/hadoop/hdfs/data

datanode上数据块的物理存储位置

dfs.replication

1

副本个数，配置默认是3,应小于datanode机器数量

5)，配置mapred-site.xml

添加如下配置：

mapreduce.framework.name

yarn

6)，配置yarn-site.xml

添加如下配置：

yarn.nodemanager.aux-services

mapreduce_shuffle

yarn.resourcemanager.webapp.address

192.168.241.128:8099

4，Hadoop启动

1)格式化namenode

$ bin/hdfs namenode –format

2)启动NameNode 和 DataNode 守护进程

$ sbin/start-dfs.sh

3)启动ResourceManager 和 NodeManager 守护进程

$ sbin/start-yarn.sh

$ cd ~/.ssh/ # 若没有该目录，请先执行一次ssh localhost

$ ssh-keygen -t rsa              # 会有提示，都按回车就可以

$ cat id_rsa.pub >> authorized_keys  # 加入授权

root@linuxidc.com:~# cd /usr/local/hadoop/

root@linuxidc.com:/usr/local/hadoop# sbin/start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

Starting namenodes on [localhost]

localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-linuxidc.com.out

localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-linuxidc.com.out

........

5，启动验证

1)执行jps命令，有如下进程，说明Hadoop正常启动

# jps

6097 NodeManager

11044 Jps

7497 -- process information unavailable

8256 Worker

5999 ResourceManager

5122 SecondaryNameNode

8106 Master

4836 NameNode

4957 DataNode