Hadoop伪分布式模式安装

环境信息

操作系统:Ubuntu 17.04 64bit

安装步骤

1. 安装配置JDK
下载JDK,链接: jdk-8u161-linux-x64.tar.gz

1.1 复制到/opt/bigdata/目录下,解压
cp ~/Downloads/jdk-8u161-linux-x64.tar.gz /opt/bigdata/
cd /opt/bigdata/
tar -xvf jdk-8u161-linux-x64.tar.gz
rm jdk-8u161-linux-x64.tar.gz
1.2 添加环境变量
打开bashrc文件
vi ~/.bashrc
在末尾添加如下命令
export JAVA_HOME=/opt/bigdata/jdk1.8.0_161
export PATH=$PATH:$JAVA_HOME/bin
生效
source ~/.bashrc
2. 安装配置ssh
安装
sudo apt-get install ssh
基于空口令生成一个新的密钥,以实现无密码登录
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
测试是否能够连接
ssh localhost

3. 安装配置Hadoop
3.1 下载
下载最新版本hadoop,链接: hadoop-3.0.0.tar.gz
3.2 解压
cp ~/Downloads/hadoop-3.0.0.tar.gz /opt/bigdata/
cd /opt/bigdata/
tar -xvf hadoop-3.0.0.tar.gz
3.3 配置
需要修改hadoop-3.0.0/etc/hadoop/目录下的4个配置文件和一个shell文件,内容如下:
core-site.xml
<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://localhost</value>
        </property>
</configuration>
hdfs-site.xml
<configuration>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
</configuration>
yarn-site.xml
<configuration>
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>localhost</value>
        </property>
        <property>
                <name>yarn.noemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
</configuration>
mapred-site.xml
<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
</configuration>
hadoop-env.sh
添加下面的命令
export JAVA_HOME=/opt/bigdata/jdk1.8.0_161

3.4 格式化HDFS文件系统
cd /opt/bigdata/hadoop-3.0.0/bin/
./hdfs namenode -format

3.5 启动hadoop
cd /opt/bigdata/hadoop-3.0.0/sbin/
./start-all.sh
看到下面输出

WARNING: Attempting to start all Apache Hadoop daemons as pxchen in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [pxchen-VirtualBox]
Starting resourcemanager
Starting nodemanagers

3.5 检查
Resource Manager
打开浏览器,输入下面地址:http://localhost:8088/
可以看到:


Name Node
输入地址:http://localhost:9870/dfshealth.html#tab-overview


参考:
1. 《Hadoop权威指南》第4版


©️2020 CSDN 皮肤主题: 大白 设计师:CSDN官方博客 返回首页