Hadoop Linux环境安装
1. 安装环境
Linux CentOS 7操作系统
JDK1.8
Hadoop 3.1.1
2. 具体步骤
2.1. 安装JDK1.8 官网下载
解压文件(命令:tar -zxvf jdk-8u191-linux-x64.tar.gz) 。
配置环境变量(命令:vi/etc/profile)在末尾添加一下配置:
export JAVA_HOME=/home/tools/jdk1.8.0_191
export JRE_HOME=/home/tools/jdk1.8.0_191/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
1
2
3
4
刷新配置文件(命令:source /etc/profile)
验证是否安装成功(命令:java -version)
2.2 安装Hadoop 官网下载 版本3.1.1
解压文件(命令:tar -zxvf hadoop-3.1.1.tar.gz -C hadoop)
配置环境变量(命令:vi/etc/profile)在末尾添加一下配置:
export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
1
2
3
4
5
6
7
8
刷新环境变量文件(命令:source /etc/profile)
Hadoop 配置文件修改(文件路径:$HADOOP_HOME/etc/hadoop)
core-site.xml
fs.default.name
hdfs://localhost:9000
1
2
3
4
5
6
hdfs-site.xml
dfs.replication
1
dfs.name.dir
file:///home/hadoop/hadoopdata/hdfs/namenode
dfs.data.dir
file:///home/hadoop/hadoopdata/hdfs/datanode
1
2
3
4
5
6
7
8
9
10
11
12
13
14
修改mapred-site.xml
mapreduce.framework.name
yarn
1
2
3
4
5
6
修改yarn-site.xml
yarn.nodemanager.aux-services
mapreduce_shuffle
1
2
3
4
5
6
格式化名称节点(命令:hdfs namenode -format)
[hadoopuser@iz9lk7cs77ry8dz:~]$ hdfs namenode -format
WARNING: /home/hadoop/hadoop/logs does not exist. Creating.
2018-11-19 23:54:29,187 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = iz9lk7cs77ry8dz/172.18.13.92
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 3.1.1
……
……
……
2018-11-19 23:54:32,353 INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/hadoopdata/hdfs/namenode/current/fsimage.ckpt_0000000000000000000 using no compression
2018-11-19 23:54:32,483 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/hadoopdata/hdfs/namenode/current/fsimage.ckpt_0000000000000000000 of size 389 bytes saved in 0 seconds .
2018-11-19 23:54:32,508 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2018-11-19 23:54:32,514 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at iz9lk7cs77ry8dz/172.18.13.92
************************************************************/
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
进入sbin文件(命令:cd $HADOOP_HOME/sbin)
启动start-dfs.sh脚本(命令: ./start-dfs.sh)
[hadoopuser@iz9lk7cs77ry8dz:~/hadoop/sbin] $./start-dfs.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [iz9lk7cs77ry8dz]
2018-11-24 21:28:19,715 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
1
2
3
4
5
启动start-yarn.sh脚本(命令:./start-yarn.sh)
[hadoopuser@iz9lk7cs77ry8dz:~/hadoop/sbin] $./start-yarn.sh
Starting resourcemanager
Starting nodemanagers
1
2
3
4
2.3 监控
按照以上步骤安装成功后,可以通过默认端口(9870)查看监控(http://127.0.0.1:9870)
至此Hadoop安装完成
---------------------
作者:弓虽长
来源:CSDN
原文:https://blog.csdn.net/zhongbangxing/article/details/84257752