sqoop导出出现问题

最新推荐文章于 2024-05-02 11:45:52 发布
最新推荐文章于 2024-05-02 11:45:52 发布
阅读量1k 收藏
点赞数
文章标签： sqoop hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_39254547/article/details/115976327
版权 
[root@master ~]# hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common-2.3.0.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> use mydb2;
OK
Time taken: 3.743 seconds
hive> show tables;
OK
tb_staff
Time taken: 0.221 seconds, Fetched: 1 row(s)
hive> drop table tb_staff;
OK
Time taken: 1.589 seconds
hive> CREATE TABLE tb_staff(
    >         id INT,
    >         `name` string,
    >         birthday  date
    > ) row format delimited
    >  fields terminated by ',';
OK
Time taken: 0.366 seconds
hive> select * from tb_staff
    > ;
OK
1       宋江    1979-08-12
2       林冲    1976-04-12
3       鲁智深  1979-08-12
4       公孙胜  1976-09-13
5       秦明    1980-07-29
6       李逵    1980-06-28
7       武松    1979-06-02
8       朱武    1975-07-06
9       黄信    1969-08-09
10      孙立    1974-07-08
Time taken: 0.18 seconds, Fetched: 10 row(s)
hive>  show create table tb_staff;
OK
CREATE TABLE `tb_staff`(
  `id` int, 
  `name` string, 
  `birthday` date)
ROW FORMAT SERDE 
  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
WITH SERDEPROPERTIES ( 
  'field.delim'=',', 
  'serialization.format'=',') 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  'hdfs://ns1/user/hive/warehouse/mydb2.db/tb_staff'
TBLPROPERTIES (
  'transient_lastDdlTime'='1618919821')
Time taken: 0.125 seconds, Fetched: 17 row(s)
hive>

因为之前已经创建了一张表，所以现在我重新删除了，上面是我重新创建了一个hive，已经插入了数据，在hdfs上面是可以下载查看。

在Linux上的MySQL上也创建了一张空表

mariadb> CREATE TABLE tb_emp_bak (
	id INT PRIMARY KEY AUTO_INCREMENT COMMENT '编号', 
	`name`  VARCHAR(20) NOT NULL COMMENT '姓名',
	birthday DATE COMMENT '生日'
);
Query OK, 0 rows affected (0.01 sec)

mariadb> SELECT * FROM tb_emp_bak;
Empty set

mariadb> DESC tb_emp_bak;
+----------+-------------+------+-----+---------+----------------+
| Field    | Type        | Null | Key | Default | Extra          |
+----------+-------------+------+-----+---------+----------------+
| id       | int(11)     | NO   | PRI | NULL    | auto_increment |
| name     | varchar(20) | NO   |     | NULL    |                |
| birthday | date        | YES  |     | NULL    |                |
+----------+-------------+------+-----+---------+----------------+
3 rows in set (0.06 sec)

mariadb>

上面所提到的一些只不过是想要说一下我觉得操作都应该是正确了。

另外还说一下虚拟机的版本是mariadb5.5.68，hadoop-2.7.6,sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

mariadb> select version();
+----------------+
| version()      |
+----------------+
| 5.5.68-MariaDB |
+----------------+
1 row in set (0.01 sec)

下面是我的sqoop中的lib目录，里面已经添加了mysql的驱动包

[root@master lib]# ll
总用量 17616
-rw-r--r-- 1 root root  224277 1月   6 18:31 ant-contrib-1.0b3.jar
-rw-r--r-- 1 root root   36455 1月   6 18:31 ant-eclipse-1.0-jvm1.2.jar
-rw-r--r-- 1 root root 1344870 1月   6 18:31 avro-1.8.1.jar
-rw-r--r-- 1 root root  186260 1月   6 18:31 avro-mapred-1.8.1-hadoop2.jar
-rw-r--r-- 1 root root   58160 1月   6 18:31 commons-codec-1.4.jar
-rw-r--r-- 1 root root  365552 1月   6 18:31 commons-compress-1.8.1.jar
-rw-r--r-- 1 root root  109043 1月   6 18:31 commons-io-1.4.jar
-rw-r--r-- 1 root root  267634 1月   6 18:31 commons-jexl-2.1.1.jar
-rw-r--r-- 1 root root  434678 1月   6 18:31 commons-lang3-3.4.jar
-rw-r--r-- 1 root root   60686 1月   6 18:31 commons-logging-1.1.1.jar
-rw-r--r-- 1 root root  437405 1月   6 18:31 hive-common-2.3.0.jar
-rw-r--r-- 1 root root  706710 1月   6 18:31 hsqldb-1.8.0.10.jar
-rw-r--r-- 1 root root   46968 1月   6 18:35 jackson-annotations-2.6.0.jar
-rw-r--r-- 1 root root  258876 1月   6 18:35 jackson-core-2.6.5.jar
-rw-r--r-- 1 root root 1171380 1月   6 18:36 jackson-databind-2.6.5.jar
-rw-r--r-- 1 root root   84697 4月  20 19:34 java-json.jar
-rw-r--r-- 1 root root 2178774 1月   6 18:31 kite-data-core-1.1.0.jar
-rw-r--r-- 1 root root 1801469 1月   6 18:31 kite-data-hive-1.1.0.jar
-rw-r--r-- 1 root root 1768012 1月   6 18:31 kite-data-mapreduce-1.1.0.jar
-rw-r--r-- 1 root root 1765905 1月   6 18:31 kite-hadoop-compatibility-1.1.0.jar
-rw-r--r-- 1 root root 1004838 4月  20 19:18 mysql-connector-java-5.1.46.jar
-rw-r--r-- 1 root root   19827 1月   6 18:31 opencsv-2.3.jar
-rw-r--r-- 1 root root   34604 1月   6 18:31 paranamer-2.7.jar
-rw-r--r-- 1 root root   53464 1月   6 18:31 parquet-avro-1.6.0.jar
-rw-r--r-- 1 root root  892808 1月   6 18:31 parquet-column-1.6.0.jar
-rw-r--r-- 1 root root   20998 1月   6 18:31 parquet-common-1.6.0.jar
-rw-r--r-- 1 root root  279012 1月   6 18:31 parquet-encoding-1.6.0.jar
-rw-r--r-- 1 root root  375618 1月   6 18:31 parquet-format-2.2.0-rc1.jar
-rw-r--r-- 1 root root   20744 1月   6 18:31 parquet-generator-1.6.0.jar
-rw-r--r-- 1 root root  205389 1月   6 18:31 parquet-hadoop-1.6.0.jar
-rw-r--r-- 1 root root 1033299 1月   6 18:31 parquet-jackson-1.6.0.jar
-rw-r--r-- 1 root root   25496 1月   6 18:31 slf4j-api-1.6.1.jar
-rw-r--r-- 1 root root  592319 1月   6 18:31 snappy-java-1.1.1.6.jar
-rw-r--r-- 1 root root   99555 1月   6 18:31 xz-1.5.jar
[root@master lib]#

还有sqoop中的sqoop-env.sh

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# included in all the hadoop scripts with source command
# should not be executable directly
# also should not be passed any arguments, since we need original $*

# Set Hadoop-specific environment variables here.

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/hadoop

#Set path to where hadoop-*-core.jar is available
#export HADOOP_MAPRED_HOME=

#set the path to where bin/hbase is available
export HBASE_HOME=/opt/sqoop/hbase-1.6.0-jars

#Set the path to where bin/hive is available
export HIVE_HOME=/opt/hive

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/opt/zk/conf
export ZOOKEEPER_HOME=/opt/zk

export HCAT_HOME=/opt/hive/hcatalog

实操过程：
第一次执行没成功


```powershell
[root@master sqoop]# sqoop export\
>  --connect 'jdbc:mysql://master:3306/sqoop?useUnicode=true&characterEncoding=utf-8'\
>  --username root\
>  --password 123456\
>  --table tb_emp_bak\
>  --input-fields-terminated-by ','\
>  --export-dir /user/hive/warehouse/mydb2.db/tb_staff/part-m-00000
Warning: /opt/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
21/04/20 20:04:06 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
21/04/20 20:04:06 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/04/20 20:04:07 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
21/04/20 20:04:07 INFO tool.CodeGenTool: Beginning code generation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/sqoop/hbase-1.6.0-jars/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/04/20 20:04:07 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tb_emp_bak` AS t LIMIT 1
21/04/20 20:04:07 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tb_emp_bak` AS t LIMIT 1
21/04/20 20:04:07 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop
注: /tmp/sqoop-root/compile/f9b507627fd47a634bdf9e7ef47fcdca/tb_emp_bak.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
21/04/20 20:04:10 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/f9b507627fd47a634bdf9e7ef47fcdca/tb_emp_bak.jar
21/04/20 20:04:10 INFO mapreduce.ExportJobBase: Beginning export of tb_emp_bak
21/04/20 20:04:11 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
21/04/20 20:04:12 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
21/04/20 20:04:12 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
21/04/20 20:04:12 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
21/04/20 20:04:16 INFO input.FileInputFormat: Total input paths to process : 1
21/04/20 20:04:16 INFO input.FileInputFormat: Total input paths to process : 1
21/04/20 20:04:16 INFO mapreduce.JobSubmitter: number of splits:4
21/04/20 20:04:16 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
21/04/20 20:04:16 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1618917590412_0010
21/04/20 20:04:17 INFO impl.YarnClientImpl: Submitted application application_1618917590412_0010
21/04/20 20:04:17 INFO mapreduce.Job: The url to track the job: http://slave1:8088/proxy/application_1618917590412_0010/
21/04/20 20:04:17 INFO mapreduce.Job: Running job: job_1618917590412_0010
21/04/20 20:04:24 INFO mapreduce.Job: Job job_1618917590412_0010 running in uber mode : false
21/04/20 20:04:24 INFO mapreduce.Job:  map 0% reduce 0%
21/04/20 20:04:32 INFO mapreduce.Job:  map 100% reduce 0%
21/04/20 20:04:33 INFO mapreduce.Job: Job job_1618917590412_0010 failed with state FAILED due to: Task failed task_1618917590412_0010_m_000003
Job failed as tasks failed. failedMaps:1 failedReduces:0

21/04/20 20:04:33 INFO mapreduce.Job: Counters: 12
        Job Counters 
                Failed map tasks=1
                Killed map tasks=3
                Launched map tasks=4
                Data-local map tasks=4
                Total time spent by all maps in occupied slots (ms)=21886
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=21886
                Total vcore-milliseconds taken by all map tasks=21886
                Total megabyte-milliseconds taken by all map tasks=22411264
        Map-Reduce Framework
                CPU time spent (ms)=0
                Physical memory (bytes) snapshot=0
                Virtual memory (bytes) snapshot=0
21/04/20 20:04:33 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
21/04/20 20:04:33 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 21.2356 seconds (0 bytes/sec)
21/04/20 20:04:33 INFO mapreduce.ExportJobBase: Exported 0 records.
21/04/20 20:04:33 ERROR mapreduce.ExportJobBase: Export job failed!
21/04/20 20:04:33 ERROR tool.ExportTool: Error during export: 
Export job failed!
        at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:445)
        at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931)
        at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80)
        at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:252)

第二次执行，修改了分隔符，因为网上说大部分是因为分隔符的问题，可是没有成功。可能是存着侥幸心理，但是我特别指定了“，”为分隔符。

[root@master sqoop]# sqoop export\
>  --connect 'jdbc:mysql://master:3306/sqoop?useUnicode=true&characterEncoding=utf-8'\
>  --username root\
>  --password 123456\
>  --table tb_emp_bak\
>  --input-fields-terminated-by '\001'\
>  --export-dir /user/hive/warehouse/mydb2.db/tb_staff/
Warning: /opt/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
21/04/20 20:31:10 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
21/04/20 20:31:10 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/04/20 20:31:10 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
21/04/20 20:31:10 INFO tool.CodeGenTool: Beginning code generation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/sqoop/hbase-1.6.0-jars/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/04/20 20:31:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tb_emp_bak` AS t LIMIT 1
21/04/20 20:31:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tb_emp_bak` AS t LIMIT 1
21/04/20 20:31:11 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop
注: /tmp/sqoop-root/compile/c2eaacaec71a04fe9aee710f89b4153f/tb_emp_bak.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
21/04/20 20:31:19 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/c2eaacaec71a04fe9aee710f89b4153f/tb_emp_bak.jar
21/04/20 20:31:19 INFO mapreduce.ExportJobBase: Beginning export of tb_emp_bak
21/04/20 20:31:20 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
21/04/20 20:31:22 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
21/04/20 20:31:22 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
21/04/20 20:31:22 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
21/04/20 20:31:29 INFO input.FileInputFormat: Total input paths to process : 2
21/04/20 20:31:29 INFO input.FileInputFormat: Total input paths to process : 2
21/04/20 20:31:29 INFO mapreduce.JobSubmitter: number of splits:3
21/04/20 20:31:29 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
21/04/20 20:31:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1618917590412_0012
21/04/20 20:31:30 INFO impl.YarnClientImpl: Submitted application application_1618917590412_0012
21/04/20 20:31:30 INFO mapreduce.Job: The url to track the job: http://slave1:8088/proxy/application_1618917590412_0012/
21/04/20 20:31:30 INFO mapreduce.Job: Running job: job_1618917590412_0012
21/04/20 20:31:41 INFO mapreduce.Job: Job job_1618917590412_0012 running in uber mode : false
21/04/20 20:31:41 INFO mapreduce.Job:  map 0% reduce 0%
21/04/20 20:31:57 INFO mapreduce.Job:  map 100% reduce 0%
21/04/20 20:31:58 INFO mapreduce.Job: Job job_1618917590412_0012 failed with state FAILED due to: Task failed task_1618917590412_0012_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

21/04/20 20:31:58 INFO mapreduce.Job: Counters: 8
        Job Counters 
                Failed map tasks=3
                Launched map tasks=3
                Data-local map tasks=3
                Total time spent by all maps in occupied slots (ms)=36131
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=36131
                Total vcore-milliseconds taken by all map tasks=36131
                Total megabyte-milliseconds taken by all map tasks=36998144
21/04/20 20:31:58 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
21/04/20 20:31:58 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 35.4898 seconds (0 bytes/sec)
21/04/20 20:31:58 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
21/04/20 20:31:58 INFO mapreduce.ExportJobBase: Exported 0 records.
21/04/20 20:31:58 ERROR mapreduce.ExportJobBase: Export job failed!
21/04/20 20:31:58 ERROR tool.ExportTool: Error during export: 
Export job failed!
        at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:445)
        at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931)
        at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80)
        at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
[root@master sqoop]#

第三次，把分隔符改回来了，虽然数据中没有空值，但还是将设置加上了。

[root@master sqoop]# sqoop export\
>  --connect 'jdbc:mysql://master:3306/sqoop?useUnicode=true&characterEncoding=utf-8'\
>  --username root\
>  --password 123456\
>  --table tb_emp_bak\
>  --input-fields-terminated-by ','\
>  --export-dir /user/hive/warehouse/mydb2.db/tb_staff/part-m-00000\
>  --input-null-string '\\N' \
>  --input-null-non-string '\\N'
Warning: /opt/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
21/04/20 20:07:01 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
21/04/20 20:07:01 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/04/20 20:07:01 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
21/04/20 20:07:01 INFO tool.CodeGenTool: Beginning code generation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/sqoop/hbase-1.6.0-jars/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/04/20 20:07:02 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tb_emp_bak` AS t LIMIT 1
21/04/20 20:07:02 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tb_emp_bak` AS t LIMIT 1
21/04/20 20:07:02 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop
注: /tmp/sqoop-root/compile/494c9dd47e075ae19fccc77d7a680fb9/tb_emp_bak.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
21/04/20 20:07:04 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/494c9dd47e075ae19fccc77d7a680fb9/tb_emp_bak.jar
21/04/20 20:07:04 INFO mapreduce.ExportJobBase: Beginning export of tb_emp_bak
21/04/20 20:07:04 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
21/04/20 20:07:05 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
21/04/20 20:07:05 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
21/04/20 20:07:05 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
21/04/20 20:07:09 INFO input.FileInputFormat: Total input paths to process : 1
21/04/20 20:07:09 INFO input.FileInputFormat: Total input paths to process : 1
21/04/20 20:07:09 INFO mapreduce.JobSubmitter: number of splits:4
21/04/20 20:07:09 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
21/04/20 20:07:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1618917590412_0011
21/04/20 20:07:09 INFO impl.YarnClientImpl: Submitted application application_1618917590412_0011
21/04/20 20:07:09 INFO mapreduce.Job: The url to track the job: http://slave1:8088/proxy/application_1618917590412_0011/
21/04/20 20:07:09 INFO mapreduce.Job: Running job: job_1618917590412_0011
21/04/20 20:07:16 INFO mapreduce.Job: Job job_1618917590412_0011 running in uber mode : false
21/04/20 20:07:16 INFO mapreduce.Job:  map 0% reduce 0%
21/04/20 20:07:25 INFO mapreduce.Job:  map 100% reduce 0%
21/04/20 20:07:26 INFO mapreduce.Job: Job job_1618917590412_0011 failed with state FAILED due to: Task failed task_1618917590412_0011_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

21/04/20 20:07:27 INFO mapreduce.Job: Counters: 12
        Job Counters 
                Failed map tasks=2
                Killed map tasks=2
                Launched map tasks=4
                Data-local map tasks=4
                Total time spent by all maps in occupied slots (ms)=25685
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=25685
                Total vcore-milliseconds taken by all map tasks=25685
                Total megabyte-milliseconds taken by all map tasks=26301440
        Map-Reduce Framework
                CPU time spent (ms)=0
                Physical memory (bytes) snapshot=0
                Virtual memory (bytes) snapshot=0
21/04/20 20:07:27 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
21/04/20 20:07:27 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 21.3258 seconds (0 bytes/sec)
21/04/20 20:07:27 INFO mapreduce.ExportJobBase: Exported 0 records.
21/04/20 20:07:27 ERROR mapreduce.ExportJobBase: Export job failed!
21/04/20 20:07:27 ERROR tool.ExportTool: Error during export: 
Export job failed!
        at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:445)
        at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931)
        at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80)
        at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:252)

第四次，加上了一个-m参数

[root@master sqoop]# sqoop export\
>  --connect 'jdbc:mysql://master:3306/sqoop?useUnicode=true&characterEncoding=utf-8'\
>  --username root\
>  --password 123456\
>  --table tb_emp_bak\
>  --input-fields-terminated-by '\001'\
>  -m 1\
>  --export-dir /user/hive/warehouse/mydb2.db/tb_staff/
Warning: /opt/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
21/04/20 20:36:31 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
21/04/20 20:36:31 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/04/20 20:36:31 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
21/04/20 20:36:31 INFO tool.CodeGenTool: Beginning code generation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/sqoop/hbase-1.6.0-jars/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/04/20 20:36:32 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tb_emp_bak` AS t LIMIT 1
21/04/20 20:36:32 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tb_emp_bak` AS t LIMIT 1
21/04/20 20:36:32 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop
注: /tmp/sqoop-root/compile/60fc6919f80c8e7f23ee1a464151bbf5/tb_emp_bak.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
21/04/20 20:36:36 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/60fc6919f80c8e7f23ee1a464151bbf5/tb_emp_bak.jar
21/04/20 20:36:36 INFO mapreduce.ExportJobBase: Beginning export of tb_emp_bak
21/04/20 20:36:37 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
21/04/20 20:36:39 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
21/04/20 20:36:39 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
21/04/20 20:36:39 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
21/04/20 20:36:45 INFO input.FileInputFormat: Total input paths to process : 2
21/04/20 20:36:45 INFO input.FileInputFormat: Total input paths to process : 2
21/04/20 20:36:45 INFO mapreduce.JobSubmitter: number of splits:1
21/04/20 20:36:45 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
21/04/20 20:36:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1618917590412_0014
21/04/20 20:36:46 INFO impl.YarnClientImpl: Submitted application application_1618917590412_0014
21/04/20 20:36:46 INFO mapreduce.Job: The url to track the job: http://slave1:8088/proxy/application_1618917590412_0014/
21/04/20 20:36:46 INFO mapreduce.Job: Running job: job_1618917590412_0014
21/04/20 20:36:57 INFO mapreduce.Job: Job job_1618917590412_0014 running in uber mode : false
21/04/20 20:36:57 INFO mapreduce.Job:  map 0% reduce 0%
21/04/20 20:37:09 INFO mapreduce.Job:  map 100% reduce 0%
21/04/20 20:37:10 INFO mapreduce.Job: Job job_1618917590412_0014 failed with state FAILED due to: Task failed task_1618917590412_0014_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

21/04/20 20:37:10 INFO mapreduce.Job: Counters: 8
        Job Counters 
                Failed map tasks=1
                Launched map tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=7716
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=7716
                Total vcore-milliseconds taken by all map tasks=7716
                Total megabyte-milliseconds taken by all map tasks=7901184
21/04/20 20:37:10 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
21/04/20 20:37:10 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 30.9485 seconds (0 bytes/sec)
21/04/20 20:37:10 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
21/04/20 20:37:10 INFO mapreduce.ExportJobBase: Exported 0 records.
21/04/20 20:37:10 ERROR mapreduce.ExportJobBase: Export job failed!
21/04/20 20:37:10 ERROR tool.ExportTool: Error during export: 
Export job failed!
        at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:445)
        at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931)
        at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80)
        at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
[root@master sqoop]#

不知道出现了什么问题

life-oss
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
重新编译打包的XmlSchema.1.1.1.jar和xfire-all1.2.6.jar
12-17
为了解决spring整合cxf,xfire遇到的jar包冲突问题，将XmlSchema.1.1.1.jar和xfire-all1.2.6.jar修改后进行了重新编译打包
ERROR: An error occurred while performing the step: “Building kernel modules“. See /var/log/nvidia-i
抓饭不吃皮牙子
11-20 6072
通过检查安装程序日志、更新内核头文件、禁用安全启动、使用 DKMS 和在相关论坛寻求帮助，在安装 NVIDIA 驱动程序并享受其提供的改善图形性能方面，你的成功机会将会增加。这些论坛由 NVIDIA 专家和社区成员负责管理，他们可以根据你特定的系统配置和安装日志的详细信息为你提供进一步的帮助。若你是一位 Linux 用户，在尝试安装或更新 NVIDIA 驱动程序时遇到了上述错误消息，不用担心，你并不孤单。在本篇博客文章中，我们将探讨这个错误的一些常见原因，并提供可能的解决方案，帮助你解决这个问题。
json-server的安装，使用，遇到的问题，解决方法以及利用axios获取数据。
BKayun的博客
07-23 2418
C:\Users\13459\AppData\Roaming\npm\cnpm.ps1，因为在此系统上禁止运行脚本。有关详细信息，请参阅 https:/ go.microsoft.com/fwlink/?LinkID=135170 中的 about_Execution_Policies。
java 编译 xlint,使用-Xlint重新编译：弃用以获取详细信息
weixin_30240263的博客
02-16 1054
I need to know how to fix these error notes:Note: Summer.java uses or overrides a deprecated API.Note: Recompile with -Xlint:deprecation for details.Here is my code:import java.util.Calendar;import ja...
\unityLibrary\src\main\java\com\unity3d\player\UnityPlayerActivity.java使用或覆盖了已过时的 API。注: 有关详细信息, 请使
弹吉他的少年郎的博客
05-16 2391
注: D:\UnityProject\UNITYTEST\Dele\Temp\gradleOut\unityLibrary\src\main\java\com\unity3d\player\UnityPlayerActivity.java使用或覆盖了已过时的 API。\unityLibrary\src\main\java\com\unity3d\player\UnityPlayerActivity.java使用或覆盖了已过时的 API。Unity原本可以愉快的使用，打包apk，如德芙般丝滑。
编译java文件提示使用或覆盖了已过时的 API，使用 -Xlint:deprecation命令重新编译，获取更详细的信息
热门推荐
hangxingkong的专栏
06-25 1万+
举个例子，假如在编译Test.java文件时，出现如下提示 注: Test.java使用或覆盖了已过时的 API。 注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。 原因：系统上安装的jdk版本较新，而你码代码过程中使用了过时的API编码规则 解决办法：使用 -Xlint:deprecation命令重新编译，获取更详细的提示信息。如有需要可查阅所安装jdk对应版本API，对过时代码进行修改 如：javac -Xlint:deprecation Test.java
命令提示符提示：请使用-Xlink:unchecked重新编译
抢街饭的专栏
11-01 4766
命令提示符提示：注意：ArrayListTest.java使用了未经检查或不安全的操作注意：要了解详细信息，请使用-Xlink:unchecked重新编译。 在jre1.6的环境中，范型的使用都会报类似警告 即支持范型的容器应该定义范型类型然后再使用，或者，在方法前加一个忽略注释： 1、ArrayList<String> al=new ArrayList<String&...
【超级详细教程】IntelliJ IDEA 从入门到上瘾，常用配置、插件、多光标操作、快捷键。
公众号：逛逛GitHub
12-17 3015
本文共计 1.5W 字，80 张图介绍 IDEA 中令人相见恨晚的技巧，本文中从入门、简单项目创建开始，介绍 IDEA 中多光标操作、常用配置、插件、版本控制等等。一定包含你在别的文章...
【Java基础】IntelliJ 使用说明
haolinbird的博客
06-15 842
存放编译后的字节码project->module先选中执行 remove, 再右键选中执行delete勾选即可使用Command+鼠标滚轮缩放字体 取消勾选 对新建类有效 触发断点需要右键选择Debug模式运行
java除去类里面的黄色警告
08-20
java除去类里面的黄色警告
sqoop导出数据到mysql故障处理---完整检查日志方法总结.txt
07-03
sqoop问题处理通用办法,首先详细描绘了一个sqoop实际问题得处理过程，然后延申到完整得hadoop日志得处理过程。非常具体实用。
3、Sqoop导出(HDFS到RMDB-mysql)
05-29
3、Sqoop导出(HDFS到RMDB-mysql) 网址：https://blog.csdn.net/chenwewi520feng/article/details/130573318 介绍通过sqoop将hdfs数据同步到关系型数据库mysql中
SQOOP导入和导出参数.pdf
08-05
sqoop操作指南
23-Sqoop数据导入导出1
08-08
2. 练习使用Sqoop将数据从Mysql中导入到HBase中 3. 练习使用Sqoop将数据在Mysql与Hive之间的导入导出 4. Sqoop上的操作举例
2、sqoop导入（RMDB-mysql、sybase到HDFS-hive）
05-29
2、sqoop导入（RMDB-mysql、sybase到HDFS-hive） 网址：https://blog.csdn.net/chenwewi520feng/article/details/130572275 介绍sqoop从关系型数据库mysql、sybase同步到hdfs、hive中
hadoop学习---基于Sqoop的文件导入导出操作
weixin_46164667的博客
05-02 821
譬如ID的范围是0-800,那么Sqoop默认运行4个进程,通过执行ISELECTMIN(id)，MAX(id) FROM emp找出id的范围,然后把4个任务的id设置范围是(0-200),(200-400)(400-600).(600-800)但请注意，这样做的前提是这些文件的结构和内容必须是一致的，即它们应该都是同一份数据的切片或者是完全相同结构的数据。6. 带有主键约束的mysql表，要注意导出的数据的主键约束的情况，不能重复。4. 导出时，mysql的表的字段数与hdfs上的列数可以不相同。
Hadoop3：集群搭建及常用命令与shell脚本整理（入门篇，从零开始搭建）
Brave_heart4pzj的博客
04-29 1049
Hadoop
node-v9.2.1-linux-x86.tar.xz
最新发布
05-06
Node.js，简称Node，是一个开源且跨平台的JavaScript运行时环境，它允许在浏览器外运行JavaScript代码。Node.js于2009年由Ryan Dahl创立，旨在创建高性能的Web服务器和网络应用程序。它基于Google Chrome的V8 JavaScript引擎，可以在Windows、Linux、Unix、Mac OS X等操作系统上运行。 Node.js的特点之一是事件驱动和非阻塞I/O模型，这使得它非常适合处理大量并发连接，从而在构建实时应用程序如在线游戏、聊天应用以及实时通讯服务时表现卓越。此外，Node.js使用了模块化的架构，通过npm（Node package manager，Node包管理器）,社区成员可以共享和复用代码，极大地促进了Node.js生态系统的发展和扩张。 Node.js不仅用于服务器端开发。随着技术的发展，它也被用于构建工具链、开发桌面应用程序、物联网设备等。Node.js能够处理文件系统、操作数据库、处理网络请求等，因此，开发者可以用JavaScript编写全栈应用程序，这一点大大提高了开发效率和便捷性。 在实践中，许多大型企业和组织已经采用Node.js作为其Web应用程序的开发平台，如Netflix、PayPal和Walmart等。它们利用Node.js提高了应用性能，简化了开发流程，并且能更快地响应市场需求。
sqoop导出hive数据时错行
07-12
对于Sqoop导出Hive数据时出现错行的问题，可能有多种原因导致。以下是一些常见的可能原因和解决方法： 1. 字段分隔符不匹配：请确保在Sqoop导出命令中指定的字段分隔符与Hive表中的分隔符一致。可以使用`--fields-...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值