修复Hadoop集群系统

修复某一台datanode

先建立免密登陆

将namenode ~/.ssh/下的公钥id_rsa.pub 内容copy 到 datanode 的 ~/.ssh/authorized_keys 里，直接追加即可

修改配置

$HADOOP_HOME/etc/slaves

如果datanode IP 变了要修改 slaves

/etc/hosts

如果datanode IP 变了,要修改 hosts映射关系。
注意，要修改所有主从节点的 /etc/hosts

10.239.47.168 hadoop91

启动hdfs

$HADOOP_HOME/sbin/
sh start-dfs.sh

启动yarn

sh start-yarn.sh

检查恢复情况

查看hadoop UI
http://10.239.47.134:50070

查看 yarn UI
http://10.239.47.134:8088

8088 为org.apache.hadoop.yarn.server.resourcemanager.ResourceManager 暴露的默认服务端口

Integration with Spark

cd $SPARK_HOME
sh sbin/start-history-server.sh

Launch hive

mysql 一般是开机自启动

# service  mysql status
Redirecting to /bin/systemctl status  mysql.service
● mysqld.service - MySQL Community Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-07-18 22:29:41 CST; 1 day 8h ago
  Process: 3233 ExecStartPost=/usr/bin/mysql-systemd-start post (code=exited, status=0/SUCCESS)
  Process: 1858 ExecStartPre=/usr/bin/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
 Main PID: 3232 (mysqld_safe)
   Memory: 178.0M
   CGroup: /system.slice/mysqld.service
           ├─3232 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
           └─3408 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mysqld.log --pid-file=/var/run/mysql...

Jul 18 22:29:28 sr531 systemd[1]: Starting MySQL Community Server...
Jul 18 22:29:37 sr531 mysqld_safe[3232]: 210718 22:29:37 mysqld_safe Logging to '/var/log/mysqld.log'.
Jul 18 22:29:37 sr531 mysqld_safe[3232]: 210718 22:29:37 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Jul 18 22:29:41 sr531 systemd[1]: Started MySQL Community Server.

cd $HIVE_HOME
nohup ./bin/hive --service metastore &