hadoop安装及简单本地模式配置

1.使用文件传输工具将jdk安装包压缩文件以及hadoop安装包压缩文件放到虚拟机指定目录/op t/install中

cd /opt/install

2.安装jdk，

tar -zxvf jdk1.8.0_241.tar.gz -C /opt/apps/java

3,配置jdk环境变量：

vi /etc/profile
export JAVA_HOME=/opt/apps/java/jdk1.8.0_241
export JRE_HOME=/${JAVA_HOME}
export CLASSPATH=.:${JAVA_HOME}/jre/lib/rt.jar:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar
export PATH=$PATH:${JAVA_HOME}/bin

4.安装hadoop

tar -zxvf hadoop-3.2.1.tar.gz -C /opt/apps/hadoop

5,配置hadoop环境变量

vi /etc/profile
export  HADOOP_HOME=/opt/apps/hadoop/hadoop-3.2.1
export  PATH=$PATH:$HADOOP_HOME/bin:$HADOOP__HOME/sbin

6.配置hadoop3.2.1/etc/hadoop/hadoop-evn.sh中的JAVA_HOME环境变量

# The java implementation to use. By default, this environment
# variable is REQUIRED on ALL platforms except OS X!
# export JAVA_HOME=/opt/apps/java/jdk1.8.0_241

7.配置core-site.xml以及hdfs-site.xml

core-site.xml

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://localhost:9000</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/opt/apps/hadoop/hadoop-3.2.1/data</value>
        </property>
</configuration>

hdfs-site.xml

<configuration>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
</configuration>

8.在根目录中的tmp目录中查看有没有hadoop-root文件如果有将该目录下的与hadoop相关的文件全部删掉，然后在/opt/apps/hadoop/hadoop-3.2.1目录下新建data目录作为数据存放位置，这个步骤可以做也可以不做，不做的话core-site.xml里面的第二项中目录改为默认目录
9，格式化namenode

hdfs namenode -format

10.启动hadoop

cd /opt/apps/hadoop/hadoop-3.2.1/sbin
start-dfs.sh

如果出现以下状况

[root@dtinone sbin]# ./start-dfs.sh 
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [dtinone]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.

分别在start-dfs.sh和stop-dfs.sh文件头部添加
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
在start-yarn.sh和stop-yarn.sh文件头部添加
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
11.验证

cd /opt/apps/hadoop/hadoop-3.2.1/sbin
start-dfs.sh
jps

查看namenode、datanode、secondarynamenode进程是否启动
12,.浏览器中打开web页面
浏览器地址栏输入http://dtinone:50070/
如果打不开，则关闭hadoop,配置hdfs-site.xml添加

<property>
        <name>dfs.namenode.http-address</name>
        <value>dtinone:50070</value>
</property>

中途遇到的其他问题
1.启动start-dfs.sh时
master: ERROR: JAVA_HOME is not set or can’t found
很坑爹，半天不知道咋回事，最后原因时在配置hadoop-evn.sh时，在export JAVA_HOME前面的#号没删除，导致配置没有生效
2.执行start-dfs.sh时
提示-bash:找不到命令
原因是：在复制环境变量时HADOOP_HOME中间的下划线变成两条，总之就在hadoop环境变量上找问题