在 Mac OSX 上安装 Hadoop

最新推荐文章于 2024-08-30 13:29:35 发布

weixin_34255793

最新推荐文章于 2024-08-30 13:29:35 发布

阅读量84

点赞数

文章标签：大数据运维 python

原文链接：https://my.oschina.net/wuliong/blog/1546247

版权

2019独角兽企业重金招聘Python工程师标准>>>

打开 System Preferences > Sharing, 在 Remote Login 前打钩，并添加用户名至 Allow access for Only these users

安装 HomeBrew

打开 Terminal，在命令行输入

brew install hadoop

hadoop 会被安装在 /usr/local/Cellar/hadoop 目录下

编辑 core-site.xml

nano /usr/local/Cellar/hadoop/2.8.1/libexec/etc/hadoop/core-site.xml

添加如下内容

<configuration>  
<property>
     <name>hadoop.tmp.dir</name>
     <value>/usr/local/Cellar/hadoop/hdfs/tmp</value>
     <description>A base for other temporary directories.</description>
</property>
<property>
     <name>fs.default.name</name>                                     
     <value>hdfs://localhost:9000</value>                             
</property>
</configuration>

编辑 mapred-site.xml

nano /usr/local/Cellar/hadoop/2.8.1/libexec/etc/hadoop/mapred-site.xml

添加如下内容

<configuration>
<property>
     <name>mapred.job.tracker</name>
     <value>localhost:9010</value>
</property>
</configuration>

编辑 hdfs-site.xml

nano /usr/local/Cellar/hadoop/2.8.1/libexec/etc/hadoop/hdfs-site.xml

添加如下内容

<configuration>
<property>
     <name>dfs.replication</name>
     <value>1</value>
</property>
</configuration>

编辑 ~/.bash_profile 添加如下内容

alias hstart="/usr/local/Cellar/hadoop/2.8.1/sbin/start-dfs.sh;/usr/local/Cellar/hadoop/2.8.1/sbin/start-yarn.sh"
alias hstop="/usr/local/Cellar/hadoop/2.8.1/sbin/stop-yarn.sh;/usr/local/Cellar/hadoop/2.8.1/sbin/stop-dfs.sh"

执行 source 命令

source ~/.bash_profile

将 /usr/local/Cellar/hadoop/2.8.1/bin 添加到 $PATH 路径上。

格式化 HDFS

hdfs namenode -format

尝试 SSH 登陆 localhost （记住自己设置的passphrase，或者最好设置为空）

ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh localhost
exit

启动 Hadoop

hstart

停止 Hadoop

hstop

Hadoop 的 Web 界面

Resource Manager: http://localhost:50070
JobTracker: http://localhost:8088
Specific Node Information: http://localhost:8042

打开 http://localhost:50070/explorer.html#/ 可查看 HDFS 文件系统，也可在命令行输入

hdfs dfs -ls /

显示当前文件系统是空的，用下面的命令新建文件夹

hdfs dfs -mkdir -p /user/<current username>

其它常用命令

$ jps
82289 ResourceManager
82386 NodeManager
83141 Jps
82036 DataNode
81946 NameNode
82156 SecondaryNameNode

$ yarn
Usage: yarn [--config confdir] [COMMAND | CLASSNAME]
  CLASSNAME                             run the class named CLASSNAME
 or
  where COMMAND is one of:
  resourcemanager                       run the ResourceManager

$ mapred
Usage: mapred [--config confdir] [--loglevel loglevel] COMMAND
       where COMMAND is one of:
  pipes                run a Pipes job
  job                  manipulate MapReduce jobs
  queue                get information regarding JobQueues

转载于:https://my.oschina.net/wuliong/blog/1546247