Hadoop学习用遇到的问题记录

1、在使用Hadoop与Hbase结合，使用BulkLoad方式导入数据时发现出现问题：Container exited with a non-zero exit code 1 Failing this attempt. Failing the application.

21/05/24 16:57:07 INFO mapreduce.Job:  map 0% reduce 0%
21/05/24 16:57:07 INFO mapreduce.Job: Job job_1620729426008_0013 failed with state FAILED due to: Application application_1620729426008_0013 failed 2 times due to AM Container for appattempt_1620729426008_0013_000002 exited with  exitCode: 1
For more detailed output, check application tracking page:http://master:8088/cluster/app/application_1620729426008_0013Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1620729426008_0013_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
	at org.apache.hadoop.util.Shell.run(Shell.java:482)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
21/05/24 16:57:07 INFO mapreduce.Job: Counters: 0

解决的方法：
通过查看hadoop中的log发现Yarn报错为:

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
java.lang.RuntimeException: java.lang.ClassNotFoundException: 
Class org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2 not found

---

2、使用HbaseTemplate 出错：
使用SpringBoot集成Hbase的过程，准备使用HBaseTemplate作为hbase的模板操作方法。将使用Template，代码如下：

@Configuration
public class HbaseConfig {

    @Value("${hbase.zookeeper.quorum}")
    private String quorum;

    @Value("${hbase.zookeeper.property.clientPort}")
    private String clientPort;

    @Value("${hbase.rootdir}")
    private String rootDir;

    @Value("${hbase.master}")
    private String master;

    @Bean
    public HbaseTemplate hbaseTemplate(){
        HbaseTemplate hbaseTemplate = new HbaseTemplate();
        org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
        conf.set("hbase.zookeeper.quorum", quorum);
        conf.set("hbase.zookeeper.property.clientPort", clientPort);
        conf.set("hbase.master", master);
        hbaseTemplate.setConfiguration(conf);
        return hbaseTemplate;
    }
}

出现错误：

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hbaseTemplate' defined in class path resource [cn/tju/seadata/config/HbaseConfig.class]: Post-processing of merged bean definition failed; nested exception is java.lang.IllegalStateException: Failed to introspect Class [org.springframework.data.hadoop.hbase.HbaseTemplate] from ClassLoader [sun.misc.Launcher$AppClassLoader@18b4aac2]

Caused by: java.lang.IllegalStateException: Failed to introspect Class [org.springframework.data.hadoop.hbase.HbaseTemplate] from ClassLoader [sun.misc.Launcher$AppClassLoader@18b4aac2]

Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/HTableInterface

可以从上面的错误中发现，出现错误的原因是HTableInterface是方法无法找到，所以报的错误。

经过查找资料发现出现错误的原因：
在Hbase 2.x 之后就已经移除了HTableInterface方法，所以2.x之后的版本就无法继续使用HbaseTemplate了！

---

3、使用Phoenix和Hbase做映射的时候遇到的问题：

0: jdbc:phoenix:localhost:2183> create view "test1" (
. . . . . . . . . . . . . . .)> "ROW" VARCHAR PRIMARY KEY,
. . . . . . . . . . . . . . .)> "cf"."name" VARCHAR  
. . . . . . . . . . . . . . .)> );
No rows affected (0.058 seconds)
0: jdbc:phoenix:localhost:2183> select * from 'test1';
Error: ERROR 601 (42P00): Syntax error. Encountered "test1" at line 1, column 15. (state=42P00,code=601)
org.apache.phoenix.exception.PhoenixParserException: ERROR 601 (42P00): Syntax error. Encountered "test1" at line 1, column 15.
	at org.apache.phoenix.exception.PhoenixParserException.newException(PhoenixParserException.java:33)
	at org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:111)
	at org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1739)
	at org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1822)
	at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1919)
	at sqlline.Commands.executeSingleQuery(Commands.java:1054)
	at sqlline.Commands.execute(Commands.java:1003)
	at sqlline.Commands.sql(Commands.java:967)
	at sqlline.SqlLine.dispatch(SqlLine.java:734)
	at sqlline.SqlLine.begin(SqlLine.java:541)
	at sqlline.SqlLine.start(SqlLine.java:267)
	at sqlline.SqlLine.main(SqlLine.java:206)
Caused by: NoViableAltException(158@[])
	at org.apache.phoenix.parse.PhoenixSQLParser.table_factor(PhoenixSQLParser.java:7581)
	at org.apache.phoenix.parse.PhoenixSQLParser.table_ref(PhoenixSQLParser.java:7468)
	at org.apache.phoenix.parse.PhoenixSQLParser.table_list(PhoenixSQLParser.java:7404)
	at org.apache.phoenix.parse.PhoenixSQLParser.parseFrom(PhoenixSQLParser.java:7369)
	at org.apache.phoenix.parse.PhoenixSQLParser.single_select(PhoenixSQLParser.java:5549)
	at org.apache.phoenix.parse.PhoenixSQLParser.unioned_selects(PhoenixSQLParser.java:5651)
	at org.apache.phoenix.parse.PhoenixSQLParser.select_node(PhoenixSQLParser.java:5717)
	at org.apache.phoenix.parse.PhoenixSQLParser.oneStatement(PhoenixSQLParser.java:848)
	at org.apache.phoenix.parse.PhoenixSQLParser.statement(PhoenixSQLParser.java:526)
	at org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:108)
	... 10 more

出错的原因：
是因为在查询语句中使用了单引号''，改成双引号""即可进行查询.

0: jdbc:phoenix:localhost:2183> select * from "test1";
+-----+-------+
| ROW | name  |
+-----+-------+
| 1   | tom   |
| 2   | honey |
+-----+-------+
2 rows selected (0.08 seconds)

---

Hbase 自定义Filter找不到类

Hbase中自定义Filter过程中出现的问题

org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: cn.tju.seadata.filter.LonLatFilter
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1609)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1154)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2965)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3299)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: java.lang.ClassNotFoundException: cn.tju.seadata.filter.LonLatFilter
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at org.apache.hadoop.hbase.util.DynamicClassLoader.tryRefreshClass(DynamicClassLoader.java:173)
	at org.apache.hadoop.hbase.util.DynamicClassLoader.loadClass(DynamicClassLoader.java:140)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1600)
	... 8 more

解决方法：生成jar包，然后将jar复制到$HBASE_HOME/lib/目录下。

同时又遇到了其他的问题：

org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.reflect.InvocationTargetException
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1609)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1154)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2965)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3299)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1605)
	... 8 more
Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException: parseFrom called on base Filter, but should be called on derived type
	at org.apache.hadoop.hbase.filter.Filter.parseFrom(Filter.java:304)
	... 13 more

产生错误的原因：
是没有重写Filter中的parseFrom包，需要自己定义一个。