Sqoop导入MySQL所有表到Hive

最新推荐文章于 2022-12-13 22:38:21 发布
最新推荐文章于 2022-12-13 22:38:21 发布
阅读量1.3k 收藏 5
点赞数 3
分类专栏： Hadoop基础教程 文章标签： sqoop hive 所有表 多表导入
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/chengyuqiang/article/details/80435294
版权

1、导入到Hive默认数据库

[root@node1 sqoop-1.4.7]# bin/sqoop-import-all-tables --connect jdbc:mysql://node1:3306/esdb --username root --password 123456 --hive-import --create-hive-table
Warning: /opt/sqoop-1.4.7/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
18/05/24 15:26:19 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
18/05/24 15:26:19 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/05/24 15:26:19 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
18/05/24 15:26:19 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
18/05/24 15:26:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/05/24 15:26:20 INFO tool.CodeGenTool: Beginning code generation
18/05/24 15:26:20 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `files` AS t LIMIT 1
18/05/24 15:26:20 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `files` AS t LIMIT 1
18/05/24 15:26:20 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.7.5
Note: /tmp/sqoop-root/compile/a38c87c70af413cbb1ec75d0b0f5ad34/files.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/05/24 15:26:22 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/a38c87c70af413cbb1ec75d0b0f5ad34/files.jar
18/05/24 15:26:22 WARN manager.MySQLManager: It looks like you are importing from mysql.
18/05/24 15:26:22 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
18/05/24 15:26:22 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
18/05/24 15:26:22 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
18/05/24 15:26:22 INFO mapreduce.ImportJobBase: Beginning import of files
18/05/24 15:26:22 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/05/24 15:26:23 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/05/24 15:26:23 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
18/05/24 15:26:26 INFO db.DBInputFormat: Using read commited transaction isolation
18/05/24 15:26:26 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `files`
18/05/24 15:26:26 INFO db.IntegerSplitter: Split size: 810; Num splits: 4 from: 1 to: 3244
18/05/24 15:26:26 INFO mapreduce.JobSubmitter: number of splits:4
18/05/24 15:26:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1526097883376_0015
18/05/24 15:26:26 INFO impl.YarnClientImpl: Submitted application application_1526097883376_0015
18/05/24 15:26:26 INFO mapreduce.Job: The url to track the job: http://bigdata03-test:8088/proxy/application_1526097883376_0015/
18/05/24 15:26:26 INFO mapreduce.Job: Running job: job_1526097883376_0015
18/05/24 15:26:33 INFO mapreduce.Job: Job job_1526097883376_0015 running in uber mode : false
18/05/24 15:26:33 INFO mapreduce.Job:  map 0% reduce 0%
18/05/24 15:26:39 INFO mapreduce.Job:  map 25% reduce 0%
18/05/24 15:26:40 INFO mapreduce.Job:  map 100% reduce 0%
18/05/24 15:26:40 INFO mapreduce.Job: Job job_1526097883376_0015 completed successfully
18/05/24 15:26:41 INFO mapreduce.Job: Counters: 31
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=570260
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=412
        HDFS: Number of bytes written=3799556
        HDFS: Number of read operations=16
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=8
    Job Counters 
        Killed map tasks=1
        Launched map tasks=4
        Other local map tasks=4
        Total time spent by all maps in occupied slots (ms)=16521
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=16521
        Total vcore-milliseconds taken by all map tasks=16521
        Total megabyte-milliseconds taken by all map tasks=16917504
    Map-Reduce Framework
        Map input records=3244
        Map output records=3244
        Input split bytes=412
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=274
        CPU time spent (ms)=6610
        Physical memory (bytes) snapshot=726429696
        Virtual memory (bytes) snapshot=8493514752
        Total committed heap usage (bytes)=437256192
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=3799556
18/05/24 15:26:41 INFO mapreduce.ImportJobBase: Transferred 3.6235 MB in 17.587 seconds (210.9803 KB/sec)
18/05/24 15:26:41 INFO mapreduce.ImportJobBase: Retrieved 3244 records.
18/05/24 15:26:41 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table files
18/05/24 15:26:41 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `files` AS t LIMIT 1
18/05/24 15:26:41 INFO hive.HiveImport: Loading uploaded data into Hive
18/05/24 15:26:43 INFO hive.HiveImport: 
18/05/24 15:26:43 INFO hive.HiveImport: Logging initialized using configuration in jar:file:/opt/hive-1.2.2/lib/hive-exec-1.2.2.jar!/hive-log4j.properties
18/05/24 15:26:46 INFO hive.HiveImport: OK
18/05/24 15:26:46 INFO hive.HiveImport: Time taken: 1.753 seconds
18/05/24 15:26:46 INFO hive.HiveImport: Loading data to table default.files
18/05/24 15:26:47 INFO hive.HiveImport: Table default.files stats: [numFiles=4, totalSize=3799556]
18/05/24 15:26:47 INFO hive.HiveImport: OK
18/05/24 15:26:47 INFO hive.HiveImport: Time taken: 0.802 seconds
18/05/24 15:26:47 INFO hive.HiveImport: Hive import complete.
18/05/24 15:26:47 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory.
18/05/24 15:26:47 INFO tool.CodeGenTool: Beginning code generation
18/05/24 15:26:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `logs` AS t LIMIT 1
18/05/24 15:26:47 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.7.5
Note: /tmp/sqoop-root/compile/a38c87c70af413cbb1ec75d0b0f5ad34/logs.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/05/24 15:26:48 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/a38c87c70af413cbb1ec75d0b0f5ad34/logs.jar
18/05/24 15:26:48 INFO mapreduce.ImportJobBase: Beginning import of logs
18/05/24 15:26:48 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
18/05/24 15:26:49 INFO db.DBInputFormat: Using read commited transaction isolation
18/05/24 15:26:49 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `logs`
18/05/24 15:26:49 INFO db.IntegerSplitter: Split size: 0; Num splits: 4 from: 1 to: 1
18/05/24 15:26:49 INFO mapreduce.JobSubmitter: number of splits:1
18/05/24 15:26:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1526097883376_0016
18/05/24 15:26:50 INFO impl.YarnClientImpl: Submitted application application_1526097883376_0016
18/05/24 15:26:50 INFO mapreduce.Job: The url to track the job: http://bigdata03-test:8088/proxy/application_1526097883376_0016/
18/05/24 15:26:50 INFO mapreduce.Job: Running job: job_1526097883376_0016
18/05/24 15:26:58 INFO mapreduce.Job: Job job_1526097883376_0016 running in uber mode : false
18/05/24 15:26:58 INFO mapreduce.Job:  map 0% reduce 0%
18/05/24 15:27:05 INFO mapreduce.Job:  map 100% reduce 0%
18/05/24 15:27:05 INFO mapreduce.Job: Job job_1526097883376_0016 completed successfully
18/05/24 15:27:05 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=218298
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=99
        HDFS: Number of bytes written=47
        HDFS: Number of read operations=4
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=4480
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=4480
        Total vcore-milliseconds taken by all map tasks=4480
        Total megabyte-milliseconds taken by all map tasks=4587520
    Map-Reduce Framework
        Map input records=1
        Map output records=1
        Input split bytes=99
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=65
        CPU time spent (ms)=1560
        Physical memory (bytes) snapshot=182292480
        Virtual memory (bytes) snapshot=2124431360
        Total committed heap usage (bytes)=112197632
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=47
18/05/24 15:27:05 INFO mapreduce.ImportJobBase: Transferred 47 bytes in 16.8987 seconds (2.7813 bytes/sec)
18/05/24 15:27:05 INFO mapreduce.ImportJobBase: Retrieved 1 records.
18/05/24 15:27:05 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table logs
18/05/24 15:27:05 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `logs` AS t LIMIT 1
18/05/24 15:27:05 WARN hive.TableDefWriter: Column day had to be cast to a less precise type in Hive
18/05/24 15:27:05 INFO hive.HiveImport: Loading uploaded data into Hive
18/05/24 15:27:07 INFO hive.HiveImport: 
18/05/24 15:27:07 INFO hive.HiveImport: Logging initialized using configuration in jar:file:/opt/hive-1.2.2/lib/hive-exec-1.2.2.jar!/hive-log4j.properties
18/05/24 15:27:10 INFO hive.HiveImport: OK
18/05/24 15:27:10 INFO hive.HiveImport: Time taken: 1.809 seconds
18/05/24 15:27:10 INFO hive.HiveImport: Loading data to table default.logs
18/05/24 15:27:11 INFO hive.HiveImport: Table default.logs stats: [numFiles=1, totalSize=47]
18/05/24 15:27:11 INFO hive.HiveImport: OK
18/05/24 15:27:11 INFO hive.HiveImport: Time taken: 0.694 seconds
18/05/24 15:27:11 INFO hive.HiveImport: Hive import complete.
18/05/24 15:27:11 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory.
[root@node1 sqoop-1.4.7]# 
hive> show tables;
OK
files
logs
t1
Time taken: 0.064 seconds, Fetched: 5 row(s)
hive> select count(*) from files;
Query ID = root_20180524152738_798fe3b4-acf1-48a1-a90a-5060fe30269a
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1526097883376_0017, Tracking URL = http://ndoe1:8088/proxy/application_1526097883376_0017/
Kill Command = /opt/hadoop-2.7.5/bin/hadoop job  -kill job_1526097883376_0017
Hadoop job information for Stage-1: number of mappers: 3; number of reducers: 1
2018-05-24 15:27:45,426 Stage-1 map = 0%,  reduce = 0%
2018-05-24 15:27:51,784 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 5.4 sec
2018-05-24 15:27:59,017 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 8.39 sec
MapReduce Total cumulative CPU time: 8 seconds 390 msec
Ended Job = job_1526097883376_0017
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 3  Reduce: 1   Cumulative CPU: 8.39 sec   HDFS Read: 3816408 HDFS Write: 5 SUCCESS
Total MapReduce CPU Time Spent: 8 seconds 390 msec
OK
3244
Time taken: 21.453 seconds, Fetched: 1 row(s)
hive>

2、导入到指定Hive数据库

默认是导入到default数据库中，如果想指定导入到某个数据库中，可以使用–hive-database参数

hive> create database test;
OK
Time taken: 0.158 seconds
hive>
[root@node1 sqoop-1.4.7]# bin/sqoop-import-all-tables --connect jdbc:mysql://node1:3306/esdb --username root --password 123456 --hive-import --hive-database test --create-hive-table
Warning: /opt/sqoop-1.4.7/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
18/05/24 16:00:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
18/05/24 16:00:06 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/05/24 16:00:06 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
18/05/24 16:00:06 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
18/05/24 16:00:06 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/05/24 16:00:06 INFO tool.CodeGenTool: Beginning code generation
18/05/24 16:00:06 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `files` AS t LIMIT 1
18/05/24 16:00:06 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `files` AS t LIMIT 1
18/05/24 16:00:06 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.7.5
Note: /tmp/sqoop-root/compile/3036daf1aa7539d081b68965b6d1b061/files.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/05/24 16:00:08 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/3036daf1aa7539d081b68965b6d1b061/files.jar
18/05/24 16:00:08 WARN manager.MySQLManager: It looks like you are importing from mysql.
18/05/24 16:00:08 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
18/05/24 16:00:08 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
18/05/24 16:00:08 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
18/05/24 16:00:08 INFO mapreduce.ImportJobBase: Beginning import of files
18/05/24 16:00:08 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/05/24 16:00:09 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/05/24 16:00:10 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
18/05/24 16:00:12 INFO db.DBInputFormat: Using read commited transaction isolation
18/05/24 16:00:12 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `files`
18/05/24 16:00:12 INFO db.IntegerSplitter: Split size: 810; Num splits: 4 from: 1 to: 3244
18/05/24 16:00:12 INFO mapreduce.JobSubmitter: number of splits:4
18/05/24 16:00:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1526097883376_0019
18/05/24 16:00:12 INFO impl.YarnClientImpl: Submitted application application_1526097883376_0019
18/05/24 16:00:12 INFO mapreduce.Job: The url to track the job: http://bigdata03-test:8088/proxy/application_1526097883376_0019/
18/05/24 16:00:12 INFO mapreduce.Job: Running job: job_1526097883376_0019
18/05/24 16:00:20 INFO mapreduce.Job: Job job_1526097883376_0019 running in uber mode : false
18/05/24 16:00:20 INFO mapreduce.Job:  map 0% reduce 0%
18/05/24 16:00:26 INFO mapreduce.Job:  map 25% reduce 0%
18/05/24 16:00:27 INFO mapreduce.Job:  map 100% reduce 0%
18/05/24 16:00:27 INFO mapreduce.Job: Job job_1526097883376_0019 completed successfully
18/05/24 16:00:28 INFO mapreduce.Job: Counters: 31
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=570260
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=412
        HDFS: Number of bytes written=3799556
        HDFS: Number of read operations=16
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=8
    Job Counters 
        Killed map tasks=1
        Launched map tasks=4
        Other local map tasks=4
        Total time spent by all maps in occupied slots (ms)=17383
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=17383
        Total vcore-milliseconds taken by all map tasks=17383
        Total megabyte-milliseconds taken by all map tasks=17800192
    Map-Reduce Framework
        Map input records=3244
        Map output records=3244
        Input split bytes=412
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=283
        CPU time spent (ms)=7450
        Physical memory (bytes) snapshot=732844032
        Virtual memory (bytes) snapshot=8491610112
        Total committed heap usage (bytes)=444596224
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=3799556
18/05/24 16:00:28 INFO mapreduce.ImportJobBase: Transferred 3.6235 MB in 18.5387 seconds (200.1492 KB/sec)
18/05/24 16:00:28 INFO mapreduce.ImportJobBase: Retrieved 3244 records.
18/05/24 16:00:28 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table files
18/05/24 16:00:28 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `files` AS t LIMIT 1
18/05/24 16:00:28 INFO hive.HiveImport: Loading uploaded data into Hive
18/05/24 16:00:30 INFO hive.HiveImport: 
18/05/24 16:00:30 INFO hive.HiveImport: Logging initialized using configuration in jar:file:/opt/hive-1.2.2/lib/hive-exec-1.2.2.jar!/hive-log4j.properties
18/05/24 16:00:33 INFO hive.HiveImport: OK
18/05/24 16:00:33 INFO hive.HiveImport: Time taken: 1.786 seconds
18/05/24 16:00:33 INFO hive.HiveImport: Loading data to table test.files
18/05/24 16:00:34 INFO hive.HiveImport: Table test.files stats: [numFiles=4, totalSize=3799556]
18/05/24 16:00:34 INFO hive.HiveImport: OK
18/05/24 16:00:34 INFO hive.HiveImport: Time taken: 0.797 seconds
18/05/24 16:00:34 INFO hive.HiveImport: Hive import complete.
18/05/24 16:00:34 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory.
18/05/24 16:00:34 INFO tool.CodeGenTool: Beginning code generation
18/05/24 16:00:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `logs` AS t LIMIT 1
18/05/24 16:00:34 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.7.5
Note: /tmp/sqoop-root/compile/3036daf1aa7539d081b68965b6d1b061/logs.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/05/24 16:00:35 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/3036daf1aa7539d081b68965b6d1b061/logs.jar
18/05/24 16:00:35 INFO mapreduce.ImportJobBase: Beginning import of logs
18/05/24 16:00:35 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
18/05/24 16:00:36 INFO db.DBInputFormat: Using read commited transaction isolation
18/05/24 16:00:36 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `logs`
18/05/24 16:00:36 INFO db.IntegerSplitter: Split size: 0; Num splits: 4 from: 1 to: 1
18/05/24 16:00:36 INFO mapreduce.JobSubmitter: number of splits:1
18/05/24 16:00:36 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1526097883376_0020
18/05/24 16:00:36 INFO impl.YarnClientImpl: Submitted application application_1526097883376_0020
18/05/24 16:00:36 INFO mapreduce.Job: The url to track the job: http://bigdata03-test:8088/proxy/application_1526097883376_0020/
18/05/24 16:00:36 INFO mapreduce.Job: Running job: job_1526097883376_0020
18/05/24 16:00:44 INFO mapreduce.Job: Job job_1526097883376_0020 running in uber mode : false
18/05/24 16:00:44 INFO mapreduce.Job:  map 0% reduce 0%
18/05/24 16:00:51 INFO mapreduce.Job:  map 100% reduce 0%
18/05/24 16:00:51 INFO mapreduce.Job: Job job_1526097883376_0020 completed successfully
18/05/24 16:00:51 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=218298
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=99
        HDFS: Number of bytes written=47
        HDFS: Number of read operations=4
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=4155
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=4155
        Total vcore-milliseconds taken by all map tasks=4155
        Total megabyte-milliseconds taken by all map tasks=4254720
    Map-Reduce Framework
        Map input records=1
        Map output records=1
        Input split bytes=99
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=60
        CPU time spent (ms)=1380
        Physical memory (bytes) snapshot=183681024
        Virtual memory (bytes) snapshot=2122481664
        Total committed heap usage (bytes)=111149056
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=47
18/05/24 16:00:51 INFO mapreduce.ImportJobBase: Transferred 47 bytes in 15.9943 seconds (2.9385 bytes/sec)
18/05/24 16:00:51 INFO mapreduce.ImportJobBase: Retrieved 1 records.
18/05/24 16:00:51 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table logs
18/05/24 16:00:51 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `logs` AS t LIMIT 1
18/05/24 16:00:51 WARN hive.TableDefWriter: Column day had to be cast to a less precise type in Hive
18/05/24 16:00:51 INFO hive.HiveImport: Loading uploaded data into Hive
18/05/24 16:00:53 INFO hive.HiveImport: 
18/05/24 16:00:53 INFO hive.HiveImport: Logging initialized using configuration in jar:file:/opt/hive-1.2.2/lib/hive-exec-1.2.2.jar!/hive-log4j.properties
18/05/24 16:00:56 INFO hive.HiveImport: OK
18/05/24 16:00:56 INFO hive.HiveImport: Time taken: 1.805 seconds
18/05/24 16:00:56 INFO hive.HiveImport: Loading data to table test.logs
18/05/24 16:00:57 INFO hive.HiveImport: Table test.logs stats: [numFiles=1, totalSize=47]
18/05/24 16:00:57 INFO hive.HiveImport: OK
18/05/24 16:00:57 INFO hive.HiveImport: Time taken: 0.735 seconds
18/05/24 16:00:57 INFO hive.HiveImport: Hive import complete.
18/05/24 16:00:57 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory.
[root@node1 sqoop-1.4.7]#
hive> use test;
OK
Time taken: 0.03 seconds
hive> show tables;
OK
files
logs
Time taken: 0.032 seconds, Fetched: 2 row(s)
hive> select count(*) from files;
Query ID = root_20180524160128_5de1a451-e3b8-4b18-8311-653a22244470
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1526097883376_0021, Tracking URL = http://bigdata03-test:8088/proxy/application_1526097883376_0021/
Kill Command = /opt/hadoop-2.7.5/bin/hadoop job  -kill job_1526097883376_0021
Hadoop job information for Stage-1: number of mappers: 3; number of reducers: 1
2018-05-24 16:01:35,411 Stage-1 map = 0%,  reduce = 0%
2018-05-24 16:01:41,652 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 5.71 sec
2018-05-24 16:01:48,945 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 8.75 sec
MapReduce Total cumulative CPU time: 8 seconds 750 msec
Ended Job = job_1526097883376_0021
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 3  Reduce: 1   Cumulative CPU: 8.75 sec   HDFS Read: 3816501 HDFS Write: 5 SUCCESS
Total MapReduce CPU Time Spent: 8 seconds 750 msec
OK
3244
Time taken: 21.33 seconds, Fetched: 1 row(s)
hive>
程裕强
关注
  • 3
    点赞
  • 踩
  • 5
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
2、sqoop导入（RMDB-mysql、sybase到HDFS-hive）
05-29
2、sqoop导入（RMDB-mysql、sybase到HDFS-hive） 网址：https://blog.csdn.net/chenwewi520feng/article/details/130572275 介绍sqoop从关系型数据库mysql、sybase同步到hdfs、hive中
sqoop将Mysql数据导入Hive中
Data & Analysis
07-24 1372
本实验Sqoop版本：Sqoop 1.4.6.2.6.0.3-8 1 mysql创建表 DROP TABLE IF EXISTS `testsqoop`; CREATE TABLE `testsqoop` ( `XH` varchar(255) CHARACTER SET utf8 NOT NULL COMMENT '序号', `SSQ` varchar(255) CHARACTER SET utf8 DEFAULT NULL COMMENT '所属区' ) ENGINE=InnoDB...
数据工具sqoop用法之mysql与hive数据导入导出
大数据技术派
12-28 1019
目录 一. Sqoop介绍 二. Mysql 数据导入到 Hive 三. Hive数据导入到Mysql 四. mysql数据增量导入hive 1. 基于递增列Append导入 1). 创建hive表 2). 创建job 3). 执行job 2.Lastmodifie...
sqoop 导入 hive
mn_kw的博客
05-27 4271
sqoop 导入 Hive 分三步： 1. 先导入--target-dir 指定的 HDFS 的目录中 2. 在 Hive 中建表 3. 调用 Hive 的 LOAD DATA INPATH 把 --target-dir中的数据移动到 Hive 中 import --hive-import --hive-table dw_hd.ods_store --connect jdbc:oracle:th...
教你使用Sqoop一次性将mysql中的十张表导入到hive中
m0_67401660的博客
07-31 417
这是一篇实验随记文章，记录笔者学习过程中做的一些实验。第八次作业将mysql中的十张表导入到大数据平台hive中。
使用Sqoop将MySQL数据导入Hive
刘李404not_found的博客
08-27 681
一、说明 MySQL数据导入Hive方法与导入HDFS类似：https://blog.csdn.net/qq_39680564/article/details/100081659，Sqoop只是将数据导入HDFS后再转成Hive表 二、Hive参数 Hive除了自己的参数，同样也支持import的参数，见导入HDFS文章 --hive-home <dir> 覆盖 $HIVE_HOM...
sqoop从mysql中导入数据到parquet格式的hive中
07-07
sqoop导入数据到hive
sqoop-spark-hive:MYSQL | SQOOP | 火花| HIVE工作流程
04-27
mysql，sqoop，spark，hive工作流管道它将通过两个表order和order_items产生每日收入Sqoop 以“ parquet”格式将订单和order_items数据从mysql导入到hdfs 在这里，我们正在创建sqoop作业以用于增量数据导入命令订单...
实战Sqoop数据导入及大数据用户行为案例分析
02-20
实战Sqoop数据导入及大数据用户行为案例分析...08_Sqoop将MySQL数据导入Hive表中 09_Sqoop的导出及脚本中使用的方式 10_案例分析-动态分区的实现 11_案例分析-源表的分区加载创建 12_案例分析-指标分析使用Sqoop导出
airflow_demo:使用Sqoop从Mysql向Hive导入增量数据的Airflow脚本
05-14
代码演示，如何编写基本的Airflow以实现从Mysql到Hive的增量导入。 ＃问题陈述：-MySQL具有名为'employee_profile'的表，该表具有雇员信息，包括名字，姓氏和SSN。 脚本应检查表中是否有新记录和修改过的记录，并...
2.sqoop全量数据导入hive表
飞少fly的博客
06-02 1059
1.导入关系表到Hive中(需要手动创建好hive表) (1) 将我们mysql表当中的数据直接导入到hive表中的话，需要将hive的一个叫做hive-exec-1.2.2.jar包拷贝到sqoop的lib目录下 cp /opt/bigdata/hive-1.2.2/lib/hive-exec-1.2.2.jar /opt/bigdata/sqoop-1.4.7.bin__hadoop-2.6.0/lib/ (2) 准备hive数据库与表, 在hive中创建一个数据库和表 create d.
利用Sqoop将多种关系型数据库表导入Hive(不断更新)
进击的强强
10-10 1951
sqoop将mysql 和oracle数据导入hive中
Sqoop导入关系数据库到Hive
u011250186的博客
08-21 1893
存储，学习，分享 Sqoop 是 apache 下用于 RDBMS 和 HDFS 互相导数据的工具。本文以 mysql 数据库为例，实现关系数据库导入到 hdfs 和 hive。 1. 安装 Sqoop 使用 rpm 安装即可。 yum install sqoop sqoop-metastore -y 安装完之后需要下载 mysql jar 包到 sqoop 的 lib 目录。 这里使用 hive 的 metastore 的 mysql 数据库作为关系数据库，以 TBLS 表为...
sqoop导入导出表/数据到Hive
ZeroTeam_麒麟
04-22 6017
笔记: 将关系型数据库表及表中的数据复制到hive中 sqoop import : RMDBS——>hive 语法： sqoop import --connect jdbc:mysql://IP:PORT/database --username root --password PWD --table tablename --hive-import --hive-table hivetabl
sqoop导入mysql数据到Hive的各种方案
海边无限的专栏
12-13 4816
sqoop 导入到处 mysql-hive 的案例
Sqoop导入所有一个库的所有表
阿正的博客
04-08 1931
参考：Sqoop导入所有表https://www.yiibai.com/sqoop/sqoop_import_all_tables.html https://www.jianshu.com/p/da2a073f279d利用sqoop实现mysql与Hdfs间的数据的导入导出 sqoop-import-all-tableshttps://www.liangzl.com/get-articl...
Sqoop全量导入mysql表数据到HDFS
m0_63086929的博客
10-21 2447
Sqoop全量导入mysql表数据到HDFS
Sqoop导入mysql所有表到HDFS
程裕强的专栏
05-24 1525
1、sqoop-import-all-tables导入多表 [root@node1 sqoop-1.4.7]# bin/sqoop-import-all-tables --connect jdbc:mysql://node1:3306/esdb --username root --password 123456 --as-textfile --warehouse-dir /user/root W...
怎么使用sqoop将mysql数据表导入hive中
最新发布
05-13
要使用 Sqoop 将 MySQL 数据表导入到 Hive 中，需要遵循以下步骤： 1. 确保已经安装了 Sqoop 和 Hive。 2. 在 Hive 中创建一个数据库，用于存储导入的数据表。 3. 在 HDFS 中创建一个目录，用于存储导入的数据。 4. 使用 Sqoop 的 import 命令将 MySQL 数据表导入到 HDFS 中： ``` sqoop import \ --connect jdbc:mysql://mysql-server:3306/mydatabase \ --username mysqluser \ --password mysqlpassword \ --table mytable \ --target-dir /user/hive/warehouse/mydatabase.db/mytable \ --fields-terminated-by ',' \ --hive-import \ --hive-table mydatabase.mytable ``` 其中，`jdbc:mysql://mysql-server:3306/mydatabase` 是 MySQL 数据库的连接 URL，`mysqluser` 和 `mysqlpassword` 分别是 MySQL 数据库的用户名和密码，`mytable` 是要导入的数据表名，`/user/hive/warehouse/mydatabase.db/mytable` 是数据导入到 HDFS 中的目录，`--fields-terminated-by ','` 指定了字段分隔符为逗号，`--hive-import` 表示需要将数据导入到 Hive 中，`--hive-table mydatabase.mytable` 指定了在 Hive 中创建的目标表的名称和所在的数据库。 5. 在 Hive 中创建一个外部表，将 HDFS 目录中的数据映射为 Hive 表： ``` CREATE EXTERNAL TABLE mydatabase.mytable ( column1 datatype1, column2 datatype2, ... ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION '/user/hive/warehouse/mydatabase.db/mytable'; ``` 注意，这里的表结构要与 MySQL 数据表的结构保持一致。 6. 使用 Hive 的 INSERT INTO 语句将数据从外部表中插入到 Hive 表中： ``` INSERT INTO mydatabase.mytable SELECT * FROM mydatabase.mytable; ``` 这样就完成了将 MySQL 数据表导入到 Hive 中的操作。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值