在google cloud建立hadoop，JAVA安装，伪分布模式

参考链接

一开始看的这个
https://blog.csdn.net/kh896424665/article/details/78765175
结果发现第一个教程有的设置defaul.sh的脚本有点问题，然后参考了下面印度风味的教程
https://docs.google.com/document/d/1CJ2hxxz8KHHd38FmXePNq3TqVtMqYKrOjL2a3356N1Y/edit
搞了两天，OK了

安装JAVA

去ORACLE下载一个，现在各个APT PPA的仓库再也不能用了，因为LICENCE的问题
https://www.oracle.com/java/technologies/javase-downloads.html

搞的这个，弄个64位的linux的tar.gz

顺便一说，可以用wget --user-name=XXX --user-password=XXX来设置登录的账号密码，这样就可以直接用云主机来下载这个软件包，不需要用terminal机去搞这个东东

修改sudo vim ~/.bashrc

export JAVA_HOME=/usr/java/jdk1.8.0_221
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar 
export PATH=$JAVA_HOME/bin:$PATH

source ~/.bashrc

修改sudo vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_221
export JAVA_BIN=$JAVA_HOME/bin
export JAVA_LIB=$JAVA_HOME/lib
export CLASSPATH=.:$JAVA_LIB/tools.jar:$JAVA_LIB/dt.jar 
export PATH=$JAVA_HOME/bin:$PATH

修改sudo vim /etc/environment

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/java/jdk1.8.0_221/lib:/usr/java/jdk1.8.0_221"  

source /etc/environment
然后check下：
java -version

免密码登录

ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh localhost

安装hadoop

下载

wget http://archive.apache.org/dist/hadoop/common/hadoop-2.7.4/hadoop-2.7.4.tar.gz
sudo tar zxvf hadoop-2.7.4.tar.gz -C /usr/local
cd /usr/local
sudo mv hadoop-2.7.4 hadoop 
sudo chmod 777 -R /usr/local/hadoop

设置启动项bashrc

sudo vi .bashrc

export JAVA_HOME=/usr/java/jdk1.8.0_221
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
export PATH=$PATH:$HADOOP_CONF_DIR/bin
export PATH=$PATH:$YARN_CONF_DIR/sbin

source ~/.bashrc
修改
vim /home/abisree28/hadoop/libexec/hadoop-config.sh
那个傻逼教程TMD就是这里没写

export JAVA_HOME=/usr/java/jdk1.8.0_221

然后修改
sudo vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh
这个就是主机的环境设置

export JAVA_HOME=/usr/java/jdk1.8.0_221
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"

修改
sudo vim /usr/local/hadoop/etc/hadoop/yarn-env.sh
我女朋友说要解释一下，这个就是从机的环境设置
那个傻逼教程，就是丛集的环境设置没写清楚，发生了一点问题

export JAVA_HOME=/usr/java/jdk1.8.0_221
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/bin

修改
sudo vim /usr/local/hadoop/etc/hadoop/core-site.xml

<configuration>
        <property>
             <name>hadoop.tmp.dir</name>
             <value>file:/usr/local/hadoop/tmp</value>
             <description>Abase for other temporary directories.</description>
        </property>
        <property>
             <name>fs.defaultFS</name>
             <value>hdfs://localhost:9000</value>
        </property>
</configuration>

修改
sudo vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml

<configuration>
        <property>
             <name>dfs.replication</name>
             <value>1</value>
        </property>
        <property>
             <name>dfs.namenode.name.dir</name>
             <value>file:/usr/local/hadoop/tmp/dfs/name</value>
        </property>
        <property>
             <name>dfs.datanode.data.dir</name>
             <value>file:/usr/local/hadoop/tmp/dfs/data</value>
        </property>
</configuration>

修改
sudo vim /usr/local/hadoop/etc/hadoop/yarn-site.xml

<configuration> 
<!-- Site specific YARN configuration properties -->
    <property> 
        <name>yarn.nodemanager.aux-services</name> 
        <value>mapreduce_shuffle</value> 
    </property> 
    <property> 
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> 
        <value>org.apache.hadoop.mapred.ShuffleHandler</value> 
    </property> 
    <property> 
        <name>yarn.resourcemanager.address</name> 
        <value>127.0.0.1:8032</value> 
    </property> 
    <property> 
        <name>yarn.resourcemanager.scheduler.address</name> 
        <value>127.0.0.1:8030</value> 
    </property> 
    <property> 
        <name>yarn.resourcemanager.resource-tracker.address</name> 
        <value>127.0.0.1:8031</value> 
    </property> 
</configuration>

sudo reboot
hadoop version
hdfs namenode -format
start-all.sh
jps