Mac搭建Hadoop

安装hadoop

配置ssh

ssh-keygen -t rsa

授权你的公钥到本地可以避免每次登陆时都要询问你的密码,
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

sudo systemsetup -f -setremotelogin on 报错setremotelogin: Turning Remote Login on or off requires Full Disk Access privileges.

下载hadoop

https://mirrors.bfsu.edu.cn/apache/hadoop/common/hadoop-2.9.2/hadoop-2.9.2.tar.gz

修改配置

core-site.xml

<configuration>
	<property>
		<name>fs.defaultFS</name>
		<value>hdfs://localhost:9000</value>
	</property>
</configuration>

hdfs-site.xml

配置存入临时文件的目录，默认目录在 /tmp 目录下，但如果重启会被删除。因此修改目录存放位置

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/Users/fanshaorong/data</value>
  </property>
</configuration>

mapred-site.xml(文件不存在，复制mapred-site.xml.template文件修改)

<configuration>
   <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
   </property>
</configuration>

yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
	<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

配置HADOOP_HOME

vi ~/.bash_profile

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_261.jdk/Contents/Home
export CLASSPAHT=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export M2_HOME=/Users/fanshaorong/Desktop/Program/apache-maven-3.6.3
export HADOOP_HOME=/Users/fanshaorong/Desktop/Program/hadoop-2.9.2
export PATH=$PATH:$JAVA_HOME/bin:$M2_HOME/bin:$HADOOP_HOME/bin

格式化文件系统

bin/hdfs namenode -format

启动NameNode和DataNode的守护进程

sbin/start-dfs.sh

启动ResourceManager和NodeManager的守护进程

sbin/start-yarn.sh

访问localhost:50070

localhost:8088