hadoop 单机版部署

Hadoop 学习

Mac Hadoop 单机版部署

1. 确定以安装 brew

2. 使用brew 安装 hadoop

下载有点慢，可以配置 brew 源,一般 brew 安装的软件默认安装在 /usr/local/Cellar 目录下

brew install hadoop 

3. 配置SSH localhost

设置免密登录 localhost

ssh-keygen -t rsa

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

ssh localhost // 测试是否成功

4. 修改hadoop配置文件

配置文件路径：/usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop

core-site.xml

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/Cellar/hadoop/hdfs/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

mapred-site.xml

<configuration>
    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:9010</value>
    </property>
</configuration>

hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.http.address</name>
        <value>0.0.0.0:50070</value>
    </property>
</configuration>

5. 运行前需先格式化

执行路径：/usr/local/Cellar/hadoop/3.1.1/bin

./hdfs namenode -format

启动路径：/usr/local/Cellar/hadoop/3.1.1/sbin

./start-dfs.sh  #启动HDFS
./stop-dfs.sh  #停止HDFS，当然不是现在停止啦

6. 查看启动情况

http://localhost:50070

$ jps

5699 ResourceManager
6019 Jps
5508 SecondaryNameNode
5797 NodeManager
5270 NameNode
32380 
5373 DataNode