大数据学习之路01——让Hadoop在MacOS上跑起来

2019-05-15 | 大数据学习之路系列01

本安装文档是在MacOS中安装单机版Hadoop。

安装目录

WZB-MacBook:50_bigdata wangzhibin$ pwd
/Users/wangzhibin/00_dev_suite/50_bigdata

准备工作

JDK

Mac安装JDK的过程略，参考：MAC下安装多版本JDK和切换几种方式

WZB-MacBook:50_bigdata wangzhibin$ java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
WZB-MacBook:50_bigdata wangzhibin$ echo $JAVA_HOME
/Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home

下载Hadoop

brew install wget
wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/core/hadoop-2.8.4/hadoop-2.8.4.tar.gz
WZB-MacBook:50_bigdata wangzhibin$ tar -zxvf hadoop-2.8.4.tar.gz

安装与配置Hadoop

修改JDK配置

WZB-MacBook:hadoop-2.8.4 wangzhibin$ vi etc/hadoop/hadoop-env.sh
export JAVA_HOME=${JAVA_HOME}改为
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home

验证Hadoop

WZB-MacBook:hadoop-2.8.4 wangzhibin$ bin/hadoop
Usage: hadoop [--config confdir] [COMMAND | CLASSNAME]
  CLASSNAME            run the class named CLASSNAME
 or
  where COMMAND is one of:
  fs                   run a generic filesystem user client
  version              print the version
  jar <jar>            run a jar file
                       note: please use "yarn jar" to launch
                             YARN applications, not this command.
  checknative [-a|-h]  check native hadoop and compression libraries availability
  distcp <srcurl> <desturl> copy file or directories recursively
  archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
  classpath            prints the class path needed to get the
  credential           interact with credential providers
                       Hadoop jar and the required libraries
  daemonlog            get/set the log level for each daemon
  trace                view and modify Hadoop tracing settings

Most commands print help when invoked w/o parameters.

单机模式执行

  $ mkdir input
  $ cp etc/hadoop/*.xml input
  $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.4.jar grep input output 'dfs[a-z.]+'
  $ cat output/*
  1	dfsadmin

配置core-site.xml

WZB-MacBook:hadoop-2.8.4 wangzhibin$ mkdir -p hdfs/tmp
WZB-MacBook:hadoop-2.8.4 wangzhibin$ vi etc/hadoop/core-site.xml

增加如下配置：

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/Users/wangzhibin/00_dev_suite/50_bigdata/hadoop-2.8.4/hdfs/t