hadoop简易安装
1.创建用户
[root@hadoop105 ~]# useradd hadoop
[root@hadoop105 ~]# passwd hadoop
更改用户 hadoop 的密码 。
新的 密码:
重新输入新的 密码:
passwd:所有的身份验证令牌已经成功更新。
2.修改用户权限
#这样hadoop可以应用sudo 指令
[root@hadoop105 ~]# vim /etc/sudoers
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-pYZXJINN-1686396682838)(images/image-20230607154133980.png)]
3.修改主机名
[root@hadoop105 ~]# vim /etc/hostname
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ciNDAOfS-1686396682838)(images/image-20230607154416578.png)]
4.切换hadoop用户并创建一些文件夹
[root@hadoop105 ~]# su hadoop
[hadoop@hadoop105 root]$ cd /home/hadoop/
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-9hg1P0wl-1686396682839)(images/image-20230607154737020.png)]
5.修改自己的环境
5.1将压缩包移动到software目录下
[hadoop@hadoop105 software]$ pwd
/home/hadoop/software
[hadoop@hadoop105 software]$ ll
总用量 650424
-rw-r--r--. 1 hadoop hadoop 195013152 6月 5 12:14 jdk-8u212-linux-x64.tar.gz
-rw-rw-r--. 1 hadoop hadoop 338075860 6月 5 11:23 hadoop-3.1.3.tar.gz
5.2解压软件包到app目录下
[hadoop@hadoop105 software]$ tar -zxvf hadoop-3.1.3.tar.gz -C /home/hadoop/app
[hadoop@hadoop105 software]$ tar -zxvf jdk-8u212-linux-x64.tar.gz -C /home/hadoop/app
5.3配置个人环境
[hadoop@hadoop105 ~]$ vim .bashrc
#将下面文字添加进去
#环境HADOOP_HOME
export HADOOP_HOME=/home/hadoop/app/hadoop-3.1.3
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH
#JAVA_HOME
export JAVA_HOME=/home/hadoop/app/jdk1.8.0_212
export PATH=$PATH:$JAVA_HOME/bin
6.修改配置文件
6.1切换到文件目录
[hadoop@hadoop105 hadoop]$ pwd
/home/hadoop/app/hadoop-3.1.3/etc/hadoop
[hadoop@hadoop105 hadoop]$ ls
capacity-scheduler.xml hadoop-policy.xml kms-acls.xml mapred-queues.xml.template yarn-env.cmd
configuration.xsl hadoop-user-functions.sh.example kms-env.sh mapred-site.xml yarn-env.sh
container-executor.cfg hdfs-site.xml kms-log4j.properties shellprofile.d yarnservice-log4j.properties
core-site.xml httpfs-env.sh kms-site.xml ssl-client.xml.example yarn-site.xml
hadoop-env.cmd httpfs-log4j.properties log4j.properties ssl-server.xml.example
hadoop-env.sh httpfs-signature.secret mapred-env.cmd user_ec_policies.xml.template
hadoop-metrics2.properties httpfs-site.xml mapred-env.sh workers
6.2修改core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.defaultFS</name>
<!-- 写自己的主机名-->
<value>hdfs://localhost:9000</value>
</property>
</configuration>
6.3修改hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
6.4修改mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>
6.5修改yarn-site.xml
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
</property>
</configuration>
6.6修改workers
#自己的主机名
vim workers
7.启动hadoop
7.1设置密钥
$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
7.2启动
如果启动失败删除掉data和logs文件,在重新格式化
#格式化
hdfs namenode -format
#启动hdfs
start-dfs.sh
#启动yarn
start-yarn.sh
#启动成功
[hadoop@hadoop105 ~]$ jps
14192 Jps
12883 SecondaryNameNode
12692 DataNode
13110 ResourceManager
13238 NodeManager
12540 NameNode
7.3web界面
需要关闭防火墙或者暴露端口
#hdfs的web界面 localhost为自己虚拟机的ip
http://localhost:9528/
#yarn的web界面 localhost为自己虚拟机的ip
http://localhost:8088/
3238 NodeManager
12540 NameNode
### 7.3web界面
需要关闭防火墙或者暴露端口
```shell
#hdfs的web界面 localhost为自己虚拟机的ip
http://localhost:9528/
#yarn的web界面 localhost为自己虚拟机的ip
http://localhost:8088/