hadoop兼容性验证

前言

Hadoop是一个由Apache基金会所开发的分布式系统基础架构，主要解决海量数据的存储和海量数据的分析计算问题，广义上来说，Hadoop通常是指一个更广泛的概念–hadoop生态圈

Hadoop优缺点：

• 优点：
  1、高可靠性：Hadoop底层维护多个数据版本，所以即使Hadoop某个计算元素或者存储出现故障，也不会导致数据的丢失
  2、高扩展性：在集群间分配任务数据，可方便的扩展到数以千计的节点上
  3、高效性：在MapReduce的思想下，Hadoop是并行工作的，以加快任务处理速度
  4、高容错性：能够自动将失败的任务重新分配

• 缺点：
  1、不适合低延时数据访问：毫秒级的数据访问
  2、无法高效对大量小文件进行存储：存储大量小文件的话，会占用NameNode大量的内存来存储文件目录和块信息，NameNode的内存总是有限的，小文件的存储的寻址时间会超过读取时间，违反了HDFS的设计目标
  3、不支持并发写入、文件随机修改：一个文件只能有一个写，不允许多个线程同时写；仅支持数据追加，不支持文件的随机修改

参考链接：
https://blog.csdn.net/weixin_43842853/article/details/123007306
https://blog.csdn.net/weixin_52112640/article/details/124907147

一、安装启动

配置java环境
yum install java-1.8.0-openjdk-devel
echo export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk >> /etc/profile
source /etc/profile

创建密钥
ssh-keygen -t rsa -P ‘’ -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

下载安装包
mkdir -p /usr/local/hadoop
wget https://mirrors.sonic.net/apache/hadoop/common/hadoop-3.3.4/hadoop-3.3.4.tar.gz -P /usr/local/hadoop
cd /usr/local/hadoop
tar -xvf hadoop-3.3.4.tar.gz

#配置核心组件core-site.xml

cat <<- EOF > /usr/local/hadoop/hadoop-3.3.4/etc/hadoop/core-site.xml
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://bogon:9000</value>
    </property>
</configuration>
EOF

#配置文件系统配置文件hdfs-site.xml

cat <<- EOF > /usr/local/hadoop/hadoop-3.3.4/etc/hadoop/hdfs-site.xml
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>
EOF

#配置env定义JAVA_HOME路径

sed -i 's!# export JAVA_HOME=!export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk!'  /usr/local/hadoop/hadoop-3.3.4/etc/hadoop/hadoop-env.sh

#hadoop-3.x为了提升安全性，需要指定操作hadoop进程的用户

sed -i "2i HDFS_DATANODE_USER=root\nHDFS_DATANODE_SECURE_USER=hdfs\nHDFS_NAMENODE_USER=root\nHDFS_SECONDARYNAMENODE_USER=root"  /usr/local/hadoop/hadoop-3.3.4/sbin/start-dfs.sh
sed -i "2i YARN_RESOURCEMANAGER_USER=root\nHADOOP_SECURE_DN_USER=yarn\nYARN_NODEMANAGER_USER=root"  /usr/local/hadoop/hadoop-3.3.4/sbin/start-yarn.sh

#格式化文件系统
cd /usr/local/hadoop/hadoop-3.3.4/
bin/hdfs namenode -format
会看到类似如下的输出：

2023-03-07 16:10:47,309 INFO namenode.FSImage: Allocated new BlockPoolId: BP-512421437-10.130.0.73-1678176647285
2023-03-07 16:10:47,333 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.
2023-03-07 16:10:47,402 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2023-03-07 16:10:47,657 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 396 bytes saved in 0 seconds .
2023-03-07 16:10:47,689 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2023-03-07 16:10:47,739 INFO namenode.FSNamesystem: Stopping services started for active state
2023-03-07 16:10:47,740 INFO namenode.FSNamesystem: Stopping services started for standby state
2023-03-07 16:10:47,747 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2023-03-07 16:10:47,748 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at bogon/10.130.0.73
************************************************************/

启动服务
[root@bogon hadoop-3.3.4]# sbin/start-dfs.sh
Starting namenodes on [bogon]
Starting datanodes
Starting secondary namenodes [bogon]
2023-03-07 17:17:19,371 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

[root@bogon hadoop-3.3.4]# sbin/start-yarn.sh
Starting resourcemanager
Starting nodemanagers

二、查看进程

#如果显示SecondaryNameNode、ResourceManager、NameNode、NodeManager、DataNode 进程代表hadoop服务启动成功

[root@bogon hadoop-3.3.4]# jps
127968 NameNode
128672 ResourceManager
128110 DataNode
128816 NodeManager
128306 SecondaryNameNode
129183 Jps

web端访问
http://本机ip:9870