Hadoop3.0.1单节点安装

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/senver_wen/article/details/79562150

前言


Hadoop是一个由Apache基金会所开发的分布式系统基础架构。用户可以在不了解分布式底层细节的情况下,开发分布式程序。充分利用集群的威力进行高速运算和存储。

HDFS(Hadoop Distributed File System):一个分布式文件系统

HDFS有高容错性的特点,并且设计用来部署在低廉的(low-cost)硬件上;而且它提供高吞吐量(high throughput)来访问应用程序的数据,适合那些有着超大数据集(large data set)的应用程序。

Hadoop的框架最核心的设计就是:HDFS和MapReduce

  • HDFS为海量的数据提供了存储
  • MapReduce为海量的数据提供了计算。

前期准备


1 更新系统

保持你的Ubuntu所程序和部件都为最新版本

sudo apt-get update

2 安装vim

sudo apt-get install vim

3 安装JDK

sudo apt-get install default-jdk
java -version

4 设定SSH(Secure Shell)无密码登录

sudo apt-get install ssh
sudo apt-get install rsync
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
ll /home/<用户名>/.ssh
ll ~/.ssh
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh localhost

安装hadoop


1 下载安装Hadoop

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.0.1/hadoop-3.0.1.tar.gz
sudo tar -zxvf hadoop-3.0.1.tar.gz
sudo mv hadoop-3.0.1 /usr/local/hadoop
ll /usr/local/hadoop

2 配置Hadoop环境变量

  • 打开文件
sudo vim ~/.bashrc
  • 添加内容:
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export PATH=${JAVA_HOME}/bin:${PATH}
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar
  • 使编辑好的脚本立即生效
source ~/.bashrc

3 修改Hadoop文件

  • 修改hadoop-env.sh
sudo vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
  • 修改core-site.xml
sudo vim /usr/local/hadoop/etc/hadoop/core-site.xml
<property>
         <name>fs.defaultFS</name>
         <value>hdfs://localhost:9000</value>
</property>
  • 修改yarn-site.xml
sudo vim /usr/local/hadoop/etc/hadoop/yarn-site.xml
<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
</property>
<property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,
                HADOOP_COMMON_HOME,
                HADOOP_HDFS_HOME,
                HADOOP_CONF_DIR,
                CLASSPATH_PREPEND_DISTCACHE,
                HADOOP_YARN_HOME,
                HADOOP_MAPRED_HOME
        </value>
</property>
<property>
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>3</value>
</property>
  • 修改mapred-site
sudo vim /usr/local/hadoop/etc/hadoop/mapred-site.xml
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>
<property>
        <name>mapreduce.application.classpath</name>
        <value>/usr/local/hadoop/etc/hadoop,
                /usr/local/hadoop/share/hadoop/common/lib/*,
                /usr/local/hadoop/share/hadoop/common/*,
                /usr/local/hadoop/share/hadoop/hdfs/lib/*,
                /usr/local/hadoop/share/hadoop/hdfs/*,
                /usr/local/hadoop/share/hadoop/mapreduce/*,
                /usr/local/hadoop/share/hadoop/yarn/lib/*,
                /usr/local/hadoop/share/hadoop/yarn/*
        </value>
</property>
  • 修改hdfs-site.xml
sudo vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml
<property>
        <name>dfs.replication</name>
        <value>1</value>
</property>
<property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop/hadoop_data/hdfs/namenode</value>
</property>
<property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop/hadoop_data/hdfs/datanode</value>
</property>

4 建⽴HDFS目录

sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/namenode
sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/datanode

5 给Hadoop文件授权

sudo chown <用户名> : <组名>  -R /usr/local/hadoop

6 格式化HDFS目录

hadoop namenode -format

7 启动hadoop

start-dfs.sh
start-yarn.sh
或  
start-all.sh
  • 关闭Hadoop服务命令
stop-all.sh

8 检查进程(jps)

jps
15157 NameNode
15273 DataNode
16281 Jps
15450 SecondaryNameNode
15786 ResourceManager
15900 NodeManager

注意: 算JPS进程一共六个,少一个代表你搭建过程有问题,一般问题都是配置文件的问题,你可以通过查看日志排错!

日志位置 /usr/local/hadoop/logs

ls /usr/local/hadoop/logs
hadoop-zwg-datanode-zwg.log       hadoop-zwg-resourcemanager-zwg.log
hadoop-zwg-datanode-zwg.out       hadoop-zwg-resourcemanager-zwg.out
hadoop-zwg-datanode-zwg.out.1     hadoop-zwg-resourcemanager-zwg.out.1
hadoop-zwg-namenode-zwg.log       hadoop-zwg-secondarynamenode-zwg.log
hadoop-zwg-namenode-zwg.out       hadoop-zwg-secondarynamenode-zwg.out
hadoop-zwg-namenode-zwg.out.1     hadoop-zwg-secondarynamenode-zwg.out.1
hadoop-zwg-nodemanager-zwg.log    SecurityAuth-zwg.audit
hadoop-zwg-nodemanager-zwg.out    userlogs
hadoop-zwg-nodemanager-zwg.out.1

9 开启Hadoop Web界面

  • ResourceManager地址
http://localhost:8088/ 
  • NameNode HDFS Web地址
http://localhost:9870/


END!

没有更多推荐了,返回首页