使用Sqoop将数据从MySQL导入HBase中报Error: org.apache.hadoop.hbase.client.Put.addColumn([B[B[B)Lorg/apache/hado

最新推荐文章于 2024-01-04 13:27:31 发布

肥仔爱上可乐

最新推荐文章于 2024-01-04 13:27:31 发布

阅读量1.4k

点赞数

分类专栏： # 解决方案 HBase 文章标签： mapreduce hadoop 大数据 mysql hbase

本文链接：https://blog.csdn.net/qq_40205014/article/details/108868563

版权

解决方案 HBase 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

【Sqoop】使用Sqoop将数据从MySQL数据库导入Hbase中出现错误

问题：

20/09/29 14:32:49 INFO mapreduce.Job: Task Id : attempt_1601345268064_0009_m_000000_2, Status : FAILED Error: org.apache.hadoop.hbase.client.Put.addColumn([B[B[B)Lorg/apache/hadoop/hbase/client/Put;

操作步骤：

1. MySQL数据库中的表结构为

CREATE TABLE `user_log` (
  `log_id` bigint(20) NOT NULL AUTO_INCREMENT,
  `ip_addr` varchar(255) DEFAULT NULL,
  `log_time` datetime DEFAULT NULL,
  `title` varchar(255) DEFAULT NULL,
  `url` varchar(255) DEFAULT NULL,
  `user_id` bigint(20) DEFAULT NULL,
  PRIMARY KEY (`log_id`) USING BTREE,
  KEY `FKherb88q97dxbtcge09ii875qm` (`user_id`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 ROW_FORMAT=DYNAMIC;

2.创建表后，再去HBase中创建一个表，用于存储MySQL中的数据

create 'user_log','info'

3.执行Sqoop导入语句

sqoop import --connect jdbc:mysql://localhost:3306/ysoa --username root \
--password root --table user_log \
--hbase-table user_log \
--column-family info --hbase-create-table --hbase-row-key log_id \
--hbase-bulkload

4. 在MapReduce的过程中出现了一个ERROR，下面是主要的错误日志

20/09/29 15:10:06 INFO client.RMProxy: Connecting to ResourceManager at hadoop01/192.168.136.5:8032
Tue Sep 29 15:10:11 CST 2020 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
20/09/29 15:10:11 INFO db.DBInputFormat: Using read commited transaction isolation
20/09/29 15:10:11 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`log_id`), MAX(`log_id`) FROM `aoa_user_log`
20/09/29 15:10:11 INFO db.IntegerSplitter: Split size: 622; Num splits: 4 from: 3 to: 2491
20/09/29 15:10:12 INFO mapreduce.JobSubmitter: number of splits:4
20/09/29 15:10:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1601345268064_0011
20/09/29 15:10:12 INFO impl.YarnClientImpl: Submitted application application_1601345268064_0011
20/09/29 15:10:12 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1601345268064_0011/
20/09/29 15:10:12 INFO mapreduce.Job: Running job: job_1601345268064_0011
20/09/29 15:10:22 INFO mapreduce.Job: Job job_1601345268064_0011 running in uber mode : false
20/09/29 15:10:22 INFO mapreduce.Job:  map 0% reduce 0%
20/09/29 15:10:34 INFO mapreduce.Job:  map 25% reduce 0%
20/09/29 15:10:35 INFO mapreduce.Job:  map 50% reduce 0%
20/09/29 15:10:35 INFO mapreduce.Job: Task Id : attempt_1601345268064_0011_m_000001_0, Status : FAILED
Error: org.apache.hadoop.hbase.client.Put.addColumn([B[B[B)Lorg/apache/hadoop/hbase/client/Put;
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

20/09/29 15:10:37 INFO mapreduce.Job:  map 0% reduce 0%
20/09/29 15:10:37 INFO mapreduce.Job: Task Id : attempt_1601345268064_0011_m_000002_0, Status : FAILED
Error: org.apache.hadoop.hbase.client.Put.addColumn([B[B[B)Lorg/apache/hadoop/hbase/client/Put;
20/09/29 15:10:44 INFO mapreduce.Job:  map 25% reduce 0%
20/09/29 15:10:44 INFO mapreduce.Job: Task Id : attempt_1601345268064_0011_m_000001_1, Status : FAILED
Error: org.apache.hadoop.hbase.client.Put.addColumn([B[B[B)Lorg/apache/hadoop/hbase/client/Put;
20/09/29 15:10:45 INFO mapreduce.Job:  map 0% reduce 0%
20/09/29 15:10:50 INFO mapreduce.Job: Task Id : attempt_1601345268064_0011_m_000001_2, Status : FAILED
Error: org.apache.hadoop.hbase.client.Put.addColumn([B[B[B)Lorg/apache/hadoop/hbase/client/Put;
20/09/29 15:11:07 INFO mapreduce.Job: Task Id : attempt_1601345268064_0011_m_000000_0, Status : FAILED
Error: org.apache.hadoop.hbase.client.Put.addColumn([B[B[B)Lorg/apache/hadoop/hbase/client/Put;
20/09/29 15:11:07 INFO mapreduce.Job: Task Id : attempt_1601345268064_0011_m_000003_0, Status : FAILED
Error: org.apache.hadoop.hbase.client.Put.addColumn([B[B[B)Lorg/apache/hadoop/hbase/client/Put;
20/09/29 15:11:13 INFO mapreduce.Job: Task Id : attempt_1601345268064_0011_m_000000_1, Status : FAILED
Error: org.apache.hadoop.hbase.client.Put.addColumn([B[B[B)Lorg/apache/hadoop/hbase/client/Put;
20/09/29 15:11:23 INFO mapreduce.Job:  map 75% reduce 0%
20/09/29 15:11:24 INFO mapreduce.Job:  map 100% reduce 100%
20/09/29 15:11:25 INFO mapreduce.Job: Job job_1601345268064_0011 failed with state FAILED due to: Task failed task_1601345268064_0011_m_000001
Job failed as tasks failed. failedMaps:1 failedReduces:0

20/09/29 15:11:25 INFO mapreduce.Job: Counters: 12
	Job Counters 
		Failed map tasks=10
		Killed map tasks=1
		Launched map tasks=11
		Other local map tasks=11
		Total time spent by all maps in occupied slots (ms)=214089
		Total time spent by all reduces in occupied slots (ms)=0
		Total time spent by all map tasks (ms)=214089
		Total vcore-seconds taken by all map tasks=214089
		Total megabyte-seconds taken by all map tasks=219227136
	Map-Reduce Framework
		CPU time spent (ms)=0
		Physical memory (bytes) snapshot=0
		Virtual memory (bytes) snapshot=0
20/09/29 15:11:25 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
20/09/29 15:11:25 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 78.8771 seconds (0 bytes/sec)
20/09/29 15:11:25 INFO mapreduce.ImportJobBase: Retrieved 0 records.
20/09/29 15:11:25 ERROR tool.ImportTool: Import failed: Import job failed!

5.这一次Map的进度有到50%，之前在25%时候就直接出现了问题，关于这个问题现还没解决，下面是我操作中相关的版本

HBase 1.2.1 
Hadoop 2.7.0
Sqoop 1.4.7（1.4.6也试过）
jdk 1.8
MySQL 5.7

解决方案：

后更换了0.98.17的Hbase，重启了服务，导入成功了，可能是各个版本之间的兼容问题

在这里插入图片描述

20/09/30 09:53:16 INFO db.DBInputFormat: Using read commited transaction isolation
20/09/30 09:53:16 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `json_log_2020_09_21`
20/09/30 09:53:16 INFO db.IntegerSplitter: Split size: 3850; Num splits: 4 from: 1 to: 15402
20/09/30 09:53:16 INFO mapreduce.JobSubmitter: number of splits:4
20/09/30 09:53:16 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1601430185716_0003
20/09/30 09:53:17 INFO impl.YarnClientImpl: Submitted application application_1601430185716_0003
20/09/30 09:53:17 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1601430185716_0003/
20/09/30 09:53:17 INFO mapreduce.Job: Running job: job_1601430185716_0003
20/09/30 09:53:23 INFO mapreduce.Job: Job job_1601430185716_0003 running in uber mode : false
20/09/30 09:53:23 INFO mapreduce.Job:  map 0% reduce 0%
20/09/30 09:53:39 INFO mapreduce.Job:  map 25% reduce 0%
20/09/30 09:53:41 INFO mapreduce.Job:  map 50% reduce 0%
20/09/30 09:53:43 INFO mapreduce.Job:  map 75% reduce 0%
20/09/30 09:53:47 INFO mapreduce.Job:  map 100% reduce 0%
20/09/30 09:53:51 INFO mapreduce.Job:  map 100% reduce 76%
20/09/30 09:53:53 INFO mapreduce.Job:  map 100% reduce 100%
20/09/30 09:53:54 INFO mapreduce.Job: Job job_1601430185716_0003 completed successfully
20/09/30 09:53:55 INFO mapreduce.Job: Counters: 49

肥仔爱上可乐

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
使用Sqoop将数据从MySQL导入HBase中报Error: org.apache.hadoop.hbase.client.Put.addColumn([B[B[B)Lorg/apache/hado

【Sqoop】使用Sqoop将数据从MySQL数据库导入HHbase中出现错误出现错误：20/09/29 14:32:49 INFO mapreduce.Job: Task Id : attempt_1601345268064_0009_m_000000_2, Status : FAILED Error: org.apache.hadoop.hbase.client.Put.addColumn([B[B[B)Lorg/apache/hadoop/hbase/client/Put;操作步骤：1. M
复制链接

扫一扫