Hadoop2.7.3 安装

Hadoop2.7.3 安装

---

VirtualBox_5.18_x64
ubuntu-16.10-desktop-amd64

环境：
VirtualBox_5.18_x64
ubuntu-16.10-desktop-amd64
设置虚拟机的安装位置-E 盘
在创建虚拟机后，在存储-控制器IDE-中添加iso映像
包括Ubuntu 和 BboxGuestAdditions.iso
VirtualBox 设置虚拟机
网络连接方式 ：桥接网卡
通过添加windows的虚拟网卡 设置桥接到虚拟网卡 ，可以实现wifi链接，具体可以baidu

先完整的安装一台虚拟机后 链接复制slave虚拟机即可
安装 virtual box下的iso 用于共享文件夹

进入ubuntu后 设置root密码
sudo passwd root
sudo apt-get update
新建项目所在目录位置：
mkdir www
mkdir www/software
这样设置的缺点就是 目录名称过长，Tab提示还需要选择，不过层次分明，可以以不同的字幕开头，会方便很多
mkdir www/softwarebox

关闭防火墙 必须！！！
1 apt-get remove iptables
2 或者 iptables down ? 自己查吧
要安装的软件：
java8 jdk-8u112-linux-x64.gz //据说8会有各种问题 7可能会更方便
hadoop hadoop-2.7.3.tar.gz //2.7.1的帖子多 3的少
spark spark-2.0.1-bin-hadoop2.7.tgz
放入共享文件夹
从共享文件夹中拷贝文件
sudo cp /media/sf_share_box/* /home/leon0/www/softwarebox/

更改安装文件权限
sudo chmod -R 777 /home/leon0/www/softwarebox

tar -zxvf www/softwarebox/spark-2.0.1-bin-hadoop2.7.tgz -C www/software
tar -zxvf www/softwarebox/hadoop-2.7.3-src.tar.gz -C www/software
tar -zxvf www/softwarebox/jdk-8u112-linux-x64.gz -C www/software
tar -zxvf www/softwarebox/hadoop-2.7.3.tar.gz -C www/software

sudo gedit /etc/profile

export JAVA_HOME=/home/leon0/www/software/jdk1.8.0_112
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
export CALSSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/bin
export SPARK_HOME=/home/leon0/www/software/spark-2.0.1-bin-hadoop2.7
export HADOOP_HOME=/home/leon0/www/software/hadoop-2.7.3
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=/home/leon0/www/software/hadoop-2.7.3/lib" #指定java的library地址，否则会Warn 找不到，然后用default
export HADOOP_ROOT_LOGGER=DEBUG,console #安装成功后关掉 ，要不log太多
export PATH=$PATH:$SPARK_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

source /etc/profile
测试是否安装成功
java -version
$HADOOP_HOME

mkdir /home/leon0/www/software/hadoop-2.7.3/tmp
mkdir /home/leon0/www/software/hadoop-2.7.3/tmp/dfs
mkdir /home/leon0/www/software/hadoop-2.7.3/tmp/dfs/data
mkdir /home/leon0/www/software/hadoop-2.7.3/tmp/dfs/name

hadoop 文件夹权限！！！
sudo chmod 777 /home/leon0/www/softwarebox/hadoop-2.7.3
查看hadoop是32还是64位的 进入安装目录下的lib /native 下
file libhadoop.so.1.0.0
libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=b65d38ef07f0484f5f22d0c40408e7a50913b574, not stripped
如果是32位需要自己编译，不过maven编译过程中总有一些插件下载不下来，囧，编译过程请参考aboutyun
更改Hadoop配置文件
/www/software/hadoop-2.7.3
gedit /home/leon0/www/software/hadoop-2.7.3/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/home/leon0/www/software/jdk1.8.0_112

gedit /home/leon0/www/software/hadoop-2.7.3/etc/hadoop/core-site.xml

<configuration>
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://Spark0:9000</value>
</property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>file:/home/leno0/www/software/hadoop-2.7.3/tmp</value>
</property>

</configuration>

gedit /home/leon0/www/software/hadoop-2.7.3/etc/hadoop/hdfs-site.xml

后来 我把 hadoop的dfs文件 放在了tmp下面
tmp/dfs/name and data

<configuration>
        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>Spark0:50090</value>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>2</value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:/home/lenon0/www/software/hadoop-2.7.3/hdfs/name</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>file:/home/lenon0/www/software/hadoop-2.7.3/hdfs/data</value>
        </property>
</configuration>

gedit /home/leon0/www/software/hadoop-2.7.3/etc/hadoop/mapred-site.xml

        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>Spark0:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>Spark0:19888</value>
        </property>
        </configuration> 

gedit /home/leon0/www/software/hadoop-2.7.3/etc/hadoop/yarn-site.xml

  <configuration>
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>Spark0</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
</configuration>

gedit /home/leon0/www/software/hadoop-2.7.3/etc/hadoop/slaves
添加从节点
Spark1
Spark2

更改Spark配置文件
cp -a spark-env.sh.template spark-env.sh
gedit spark-env.sh

export JAVA_HOME=/home/leon0/www/software/jdk1.8.0_112
export SPARK_MASTER_HOST=Spark0

cp -a slaves.template slaves
gedit slaves
Spark1
Spark2

安装完后 开始复制虚拟机 选择链接复制即可
选择重新更新mac即可

查询各个节点的ifconfig
添加Ip
gedit /etc/hosts

192.168.56.104 Spark0
192.168.56.102 Spark1
192.168.56.103 Spark2
更改主机名称
sudo vim /etc/hostname
Spark1
免密码登陆

sudo apt install openssh-server

ssh-keygen -t rsa

scp authorized_keys leon0@Spark1:~/.ssh
scp authorized_keys leon0@Spark2:~/.ssh

也可以在免密码后 将hadoop拷贝过去
scp -r hadoop-2.7.3 leon0@Spark1:/home/leon0/www/software/
scp -r hadoop-2.7.3 leon0@Spark2:/home/leon0/www/software/

scp hdfs-site.xml leon0@Spark2:/home/leon0/www/software/hadoop-2.7.3/etc/hadoop/

参考
http://blog.csdn.net/SCTU_vroy/article/details/52596655
http://blog.csdn.net/sctu_vroy/article/details/52607067
http://blog.csdn.net/m1213642578/article/details/52468829?locationNum=3