hadoop格式化带来的后遗症

tips：最近生产环境的Hadoop大数据环境经常挂（天天人工重启，手放到生产的机器就会自动敲重启环境的命令，真想铲掉。。。），于是想着是不是某些组件资源不足，于是进行了格式化hadoop之路，格式五分钟，重启三小时

1. 版本先亮出来（hadoop-3.2.0 zookeeper-3.4.9 hbase-1.2.4）

格式化的过程就不说了，能走到这一步的，相信我们一样的菜，说说要命的后遗症吧。

2. 格式完成之后，hdfs的文件目录会被清除掉，可以选择重建

hadoop fs -ls /sqoop  查看dfs里面的数据文件

3. hbase里面的表及结构会被全部清除，创建新表提示旧表已存在：table already exists

1.问题出现：
在格式化NameNode后，重新运行sqoop脚本提示表不存在，于是去hbase创建表提示table already exists
2.原因： hadoop重新格式化后，hdfs上没有了数据，在hbase中新建表却提示Table already exists。虽然HDFS上和Hbase相关的东西都已经删除了。但是zookeeper保存有hbase表的地址，数据访问是通过zookeeper的地址转到hdfs上，因此需要将Zookeeper中的相应的表也删除。

破案方案如下：
进入zookeeper的bin目录：

./zkcli.sh 运行zookeeper的客户端

[hadoop@slave1 bin]$ ./zkCli.sh 
Connecting to localhost:2181
2021-12-15 10:08:39,371 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.9-1757313, built on 08/23/2016 06:50 GMT
2021-12-15 10:08:39,374 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=slave1
2021-12-15 10:08:39,375 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_131
2021-12-15 10:08:39,377 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2021-12-15 10:08:39,377 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/home/hadoop/java/jdk1.8.0_131/jre
2021-12-15 10:08:39,378 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/home/hadoop/zookeeper/zookeeper-3.4.9/bin/../build/classes:/home/hadoop/zookeeper/zookeepe
r-3.4.9/bin/../build/lib/*.jar:/home/hadoop/zookeeper/zookeeper-3.4.9/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/hadoop/zookeeper/zookeeper-3.4.9/bin/../lib/slf4j-api-1.6.1.jar:/home/hadoop/zookeeper/zookeeper-3.4.9/bin/../lib/netty-3.10.5.Final.jar:/home/hadoop/zookeeper/zookeeper-3.4.9/bin/../lib/log4j-1.2.16.jar:/home/hadoop/zookeeper/zookeeper-3.4.9/bin/../lib/jline-0.9.94.jar:/home/hadoop/zookeeper/zookeeper-3.4.9/bin/../zookeeper-3.4.9.jar:/home/hadoop/zookeeper/zookeeper-3.4.9/bin/../src/java/lib/*.jar:/home/hadoop/zookeeper/zookeeper-3.4.9/bin/../conf:.:/usr/java/jdk1.8.0_181-cloudera/lib/dt.jar:/usr/java/jdk1.8.0_181-cloudera/lib/tools.jar2021-12-15 10:08:39,378 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2021-12-15 10:08:39,379 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2021-12-15 10:08:39,379 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2021-12-15 10:08:39,379 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2021-12-15 10:08:39,379 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2021-12-15 10:08:39,379 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-693.el7.x86_64
2021-12-15 10:08:39,379 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=hadoop
2021-12-15 10:08:39,379 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/home/hadoop
2021-12-15 10:08:39,379 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/hadoop/zookeeper/zookeeper-3.4.9/bin
2021-12-15 10:08:39,381 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWat
cher@277050dcWelcome to ZooKeeper!
2021-12-15 10:08:39,409 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to 
authenticate using SASL (unknown error)JLine support is enabled
2021-12-15 10:08:39,500 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@876] - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
2021-12-15 10:08:39,510 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x1
7db79d3f12001e, negotiated timeout = 30000
WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] 

ls /hbase/table 列出hbase的所有表

[zk: localhost:2181(CONNECTED) 0] ls /hbase/table
[stockQuotes, hbase:meta, hbase:namespace, omz]

rmr /hbase/table/omz 删除对应表的缓存

[zk: localhost:2181(CONNECTED) 0] ls /hbase/table
[stockQuotes, hbase:meta, hbase:namespace, omz]
[zk: localhost:2181(CONNECTED) 1] rmr /hbase/table/omz       
[zk: localhost:2181(CONNECTED) 2] ls /hbase/table
[stockQuotes, hbase:meta, hbase:namespace]
[zk: localhost:2181(CONNECTED) 3] 

最后去hbase重新创建表，create ‘omz’, {NAME=>‘info’,VERSIONS => 1}; 发现创建成功。

4. 运行sqoop脚本发现报访问不了hbase里面的表，报错截图如下：
   

掉头发了，明明啥也没动，经过百度 stockflow github只能看见类似的报错，而我一直在纠结could not access Hbase table 这行报错，没注意下面类找不到才是引发这个问题的真正原因，事实证明千万不要钻牛角尖，要不然stockflow也救不了你，要学会和自己和解，于是我喝了半小时的水，饱了之后继续掉头发，忽然发现为啥不能当成一个普通的类找不到的bug来对待呢，对吧，少类那我就把相应的类给导入进来，继续破案。。。

5. 找到装有sqoop服务的机器，进入sqoop安装目录下面的lib目录

我的安装目录是 /home/hadoop/sqoop1/sqoop-1.4.6-cdh5.5.2

发现里面并没有报错信息上提示所需要的类所在的jar包，于是去代码里面看下有没有用过这个类，哎，还真有，直接去本地仓库copy出来放到lib目录（实在没有的话去https://mvnrepository.com/下载）

来，最后再运行一次搞到吐的sqoop脚本，熟悉的画面出现了（长叹一口真气），破案了。。。WIN+L下班