Using Sqoop 1.4.6 With Hadoop 2.7.4

最新推荐文章于 2021-04-09 02:47:37 发布
最新推荐文章于 2021-04-09 02:47:37 发布
阅读量151 收藏
点赞数
文章标签： 数据库 大数据 java
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_33788244/article/details/85071465
版权

本文主要描述Sqoop 1.4.6的安装配置以及使用。
一、安装配置
1、Sqoop安装

[hadoop@hdp01 ~]$ wget http://mirror.bit.edu.cn/apache/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
[hadoop@hdp01 ~]$ tar -xzf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
[hadoop@hdp01 ~]$ mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha /u01/sqoop
--编辑Sqoop环境变量
[hadoop@hdp01 ~]$ cd /u01/sqoop/conf
[hadoop@hdp01 conf]$ cp sqoop-env-template.sh sqoop-env.sh
[hadoop@hdp01 conf]$ vi sqoop-env.sh
export HADOOP_COMMON_HOME=/u01/hadoop
export HADOOP_MAPRED_HOME=/u01/hadoop
export HBASE_HOME=/u01/hbase
export HIVE_HOME=/u01/hive
export ZOOCFGDIR=/u01/zookeeper/conf
--注释掉configure-sqoop中的以下内容
#if [ -z "${HCAT_HOME}" ]; then
#  if [ -d "/usr/lib/hive-hcatalog" ]; then
#    HCAT_HOME=/usr/lib/hive-hcatalog
#  elif [ -d "/usr/lib/hcatalog" ]; then
#    HCAT_HOME=/usr/lib/hcatalog
#  else
#    HCAT_HOME=${SQOOP_HOME}/../hive-hcatalog
#    if [ ! -d ${HCAT_HOME} ]; then
#       HCAT_HOME=${SQOOP_HOME}/../hcatalog
#    fi
#  fi
#fi
#if [ -z "${ACCUMULO_HOME}" ]; then
#  if [ -d "/usr/lib/accumulo" ]; then
#    ACCUMULO_HOME=/usr/lib/accumulo
#  else
#    ACCUMULO_HOME=${SQOOP_HOME}/../accumulo
#  fi
#fi
## Moved to be a runtime check in sqoop.
#if [ ! -d "${HCAT_HOME}" ]; then
#  echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
#  echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
#fi
#
#if [ ! -d "${ACCUMULO_HOME}" ]; then
#  echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
#  echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
#fi
--编辑用户环境环境变量
[hadoop@hdp01 ~]$ vi .bash_profile
export SQOOP_HOME=/u01/sqoop
export SQOOP_CONF_DIR=$SQOOP_HOME/conf
export SQOOP_CLASSPATH=$SQOOP_CONF_DIR
export PATH=$PATH:$SQOOP_HOME/bin
[hadoop@hdp01 ~]$ source .bash_profile
--验证sqoop安装
[hadoop@hdp01 ~]$ sqoop version
2017-12-28 09:30:01,801 [myid:] - INFO  [main:Sqoop@92] - Running Sqoop version: 1.4.6
Sqoop 1.4.6
git commit id c0c5a81723759fa575844a0a1eae8f510fa32c25
Compiled by root on Mon Apr 27 14:38:36 CST 2015
或者运行sqoop-version
--拷贝jdbc驱动
将MySQL、PostgreSQL以及Oracle的jdbc驱动拷贝到$SQOOP_HOME/lib

二、Sqoop使用
1、Sqoop测试各个jdbc驱动连接
1.1 Sqoop与MySQL的连接

[hadoop@hdp01 bin]$ sqoop list-tables --username root -P --connect jdbc:mysql://192.168.120.92:3306/smsqw?useSSL=false
2017-12-28 09:38:19,587 [myid:] - INFO  [main:Sqoop@92] - Running Sqoop version: 1.4.6
Enter password: 
2017-12-28 09:38:23,067 [myid:] - INFO  [main:MySQLManager@69] - Preparing to use a MySQL streaming resultset.
Phone
TestPhone
history_store
tbAreaprefix
tbAreaprefix_bak
tbBill
tbBilltmp
tbCat
tbContact
tbDataPath
tbDeliverMsg
tbDeliverMsg2
tbDest
tbLocPrefix
tbMessage
tbPrice
tbReceiver
tbSSLog
tbSendState
tbSendState2
tbSmsSendState
tbTest
tbUser

1.2 Sqoop与PostgreSQL的连接

[hadoop@hdp01 ~]$ sqoop list-tables --username rhnuser -P --connect jdbc:postgresql://192.168.120.93:5432/rhndb
2017-12-28 09:40:24,842 [myid:] - INFO  [main:Sqoop@92] - Running Sqoop version: 1.4.6
Enter password: 
2017-12-28 09:40:29,775 [myid:] - INFO  [main:SqlManager@98] - Using default fetchSize of 1000
rhnservergroupmembers
rhntemplatestring
rhnservergrouptypefeature
rhnserverhistory
qrtz_fired_triggers

1.3 Sqoop与Oracle的连接

[hadoop@hdp01 ~]$ sqoop list-tables --username spwuser -P --connect jdbc:oracle:thin:@192.168.120.121:1521/rhndb --driver oracle.jdbc.driver.OracleDriver
2017-12-28 10:01:43,337 [myid:] - INFO  [main:Sqoop@92] - Running Sqoop version: 1.4.6
Enter password: 
2017-12-28 10:01:43,425 [myid:] - INFO  [main:SqlManager@98] - Using default fetchSize of 1000
rhnservergroupmembers
rhntemplatestring
rhnservergrouptypefeature
rhnserverhistory
qrtz_fired_triggers

1.4 Sqoop与Hive的连接
基于PostgreSQL在hive上创建一个名为rhnpackagefile的表，但不导入数据，后面介绍数据导入。

[hadoop@hdp01 ~]$ sqoop create-hive-table --connect jdbc:postgresql://192.168.120.93:5432/rhndb --table rhnpackagefile --username rhnuser -P --hive-database hivedb
2017-12-28 10:32:01,376 [myid:] - INFO  [main:Sqoop@92] - Running Sqoop version: 1.4.6
Enter password: 
2017-12-28 10:32:04,699 [myid:] - INFO  [main:BaseSqoopTool@1353] - Using Hive-specific delimiters for output. You can override
2017-12-28 10:32:04,699 [myid:] - INFO  [main:BaseSqoopTool@1354] - delimiters with --fields-terminated-by, etc.
2017-12-28 10:32:04,819 [myid:] - INFO  [main:SqlManager@98] - Using default fetchSize of 1000
2017-12-28 10:32:05,015 [myid:] - INFO  [main:SqlManager@757] - Executing SQL statement: SELECT t.* FROM "rhnpackagefile" AS t LIMIT 1
2017-12-28 10:32:05,674 [myid:] - INFO  [main:HiveImport@194] - Loading uploaded data into Hive
2017-12-28 10:32:09,089 [myid:] - INFO  [Thread-6:LoggingAsyncSink$LoggingThread@85] - SLF4J: Class path contains multiple SLF4J bindings.
2017-12-28 10:32:09,090 [myid:] - INFO  [Thread-6:LoggingAsyncSink$LoggingThread@85] - SLF4J: Found binding in [jar:file:/u01/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2017-12-28 10:32:09,090 [myid:] - INFO  [Thread-6:LoggingAsyncSink$LoggingThread@85] - SLF4J: Found binding in [jar:file:/u01/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2017-12-28 10:32:09,090 [myid:] - INFO  [Thread-6:LoggingAsyncSink$LoggingThread@85] - SLF4J: Found binding in [jar:file:/u01/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2017-12-28 10:32:09,091 [myid:] - INFO  [Thread-6:LoggingAsyncSink$LoggingThread@85] - SLF4J: Found binding in [jar:file:/u01/tez/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2017-12-28 10:32:09,091 [myid:] - INFO  [Thread-6:LoggingAsyncSink$LoggingThread@85] - SLF4J: Found binding in [jar:file:/u01/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2017-12-28 10:32:09,091 [myid:] - INFO  [Thread-6:LoggingAsyncSink$LoggingThread@85] - SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2017-12-28 10:32:09,095 [myid:] - INFO  [Thread-6:LoggingAsyncSink$LoggingThread@85] - SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2017-12-28 10:32:11,996 [myid:] - INFO  [Thread-6:LoggingAsyncSink$LoggingThread@85] - 
2017-12-28 10:32:11,996 [myid:] - INFO  [Thread-6:LoggingAsyncSink$LoggingThread@85] - Logging initialized using configuration in jar:file:/u01/hive/lib/hive-common-2.3.2.jar!/hive-log4j2.properties Async: true
2017-12-28 10:32:16,650 [myid:] - INFO  [Thread-6:LoggingAsyncSink$LoggingThread@85] - OK
2017-12-28 10:32:16,783 [myid:] - INFO  [Thread-6:LoggingAsyncSink$LoggingThread@85] - Time taken: 3.433 seconds
2017-12-28 10:32:17,248 [myid:] - INFO  [main:HiveImport@242] - Hive import complete.

2、数据迁移
2.1 PostgreSQL☞Hive

[hadoop@hdp01 ~]$ sqoop import --connect jdbc:postgresql://192.168.120.93:5432/rhndb --table rhnpackagefile --username rhnuser -P --fields-terminated-by ',' --hive-import --hive-database hivedb --columns package_id,capability_id,device,inode,file_mode,username,groupname,rdev,file_size,mtime,checksum_id,linkto,flags,verifyflags,lang,created,modified --split-by modified -m 4
2017-12-28 11:24:46,666 [myid:] - INFO  [main:Sqoop@92] - Running Sqoop version: 1.4.6
Enter password: 
2017-12-28 11:24:48,891 [myid:] - INFO  [main:SqlManager@98] - Using default fetchSize of 1000
2017-12-28 11:24:48,894 [myid:] - INFO  [main:CodeGenTool@92] - Beginning code generation
2017-12-28 11:24:49,091 [myid:] - INFO  [main:SqlManager@757] - Executing SQL statement: SELECT t.* FROM "rhnpackagefile" AS t LIMIT 1
2017-12-28 11:24:49,127 [myid:] - INFO  [main:CompilationManager@94] - HADOOP_MAPRED_HOME is /u01/hadoop
Note: /tmp/sqoop-hadoop/compile/ca09f6bb133fa32808220902aedc0437/rhnpackagefile.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2017-12-28 11:24:50,481 [myid:] - INFO  [main:CompilationManager@330] - Writing jar file: /tmp/sqoop-hadoop/compile/ca09f6bb133fa32808220902aedc0437/rhnpackagefile.jar
2017-12-28 11:24:50,493 [myid:] - WARN  [main:PostgresqlManager@119] - It looks like you are importing from postgresql.
2017-12-28 11:24:50,493 [myid:] - WARN  [main:PostgresqlManager@120] - This transfer can be faster! Use the --direct
2017-12-28 11:24:50,494 [myid:] - WARN  [main:PostgresqlManager@121] - option to exercise a postgresql-specific fast path.
2017-12-28 11:24:50,495 [myid:] - INFO  [main:ImportJobBase@235] - Beginning import of rhnpackagefile
2017-12-28 11:24:50,496 [myid:] - INFO  [main:Configuration@1019] - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2017-12-28 11:24:50,634 [myid:] - INFO  [main:Configuration@1019] - mapred.jar is deprecated. Instead, use mapreduce.job.jar
2017-12-28 11:24:51,160 [myid:] - INFO  [main:Configuration@1019] - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2017-12-28 11:24:51,506 [myid:] - INFO  [main:TimelineClientImpl@123] - Timeline service address: http://hdp01:8188/ws/v1/timeline/
2017-12-28 11:24:51,696 [myid:] - INFO  [main:AHSProxy@42] - Connecting to Application History server at hdp01.thinkjoy.tt/192.168.120.96:10201
2017-12-28 11:24:53,801 [myid:] - INFO  [main:DBInputFormat@192] - Using read commited transaction isolation
2017-12-28 11:24:53,805 [myid:] - INFO  [main:Configuration@1019] - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2017-12-28 11:24:53,805 [myid:] - INFO  [main:DataDrivenDBInputFormat@147] - BoundingValsQuery: SELECT MIN("modified"), MAX("modified") FROM "rhnpackagefile"
2017-12-28 11:25:14,854 [myid:] - WARN  [main:TextSplitter@64] - Generating splits for a textual index column.
2017-12-28 11:25:14,854 [myid:] - WARN  [main:TextSplitter@65] - If your database sorts in a case-insensitive order, this may result in a partial import or duplicate records.
2017-12-28 11:25:14,854 [myid:] - WARN  [main:TextSplitter@67] - You are strongly encouraged to choose an integral split column.
2017-12-28 11:25:14,903 [myid:] - INFO  [main:JobSubmitter@396] - number of splits:6
2017-12-28 11:25:14,997 [myid:] - INFO  [main:JobSubmitter@479] - Submitting tokens for job: job_1514358672274_0009
2017-12-28 11:25:15,453 [myid:] - INFO  [main:YarnClientImpl@236] - Submitted application application_1514358672274_0009
2017-12-28 11:25:15,485 [myid:] - INFO  [main:Job@1289] - The url to track the job: http://hdp01:8088/proxy/application_1514358672274_0009/
2017-12-28 11:25:15,486 [myid:] - INFO  [main:Job@1334] - Running job: job_1514358672274_0009
2017-12-28 11:25:24,763 [myid:] - INFO  [main:Job@1355] - Job job_1514358672274_0009 running in uber mode : false
2017-12-28 11:25:24,764 [myid:] - INFO  [main:Job@1362] -  map 0% reduce 0%
2017-12-28 11:26:00,465 [myid:] - INFO  [main:Job@1362] -  map 17% reduce 0%
2017-12-28 11:26:01,625 [myid:] - INFO  [main:Job@1362] -  map 50% reduce 0%
2017-12-28 11:26:03,643 [myid:] - INFO  [main:Job@1362] -  map 83% reduce 0%
2017-12-28 11:34:22,028 [myid:] - INFO  [main:Job@1362] -  map 100% reduce 0%
2017-12-28 11:34:22,035 [myid:] - INFO  [main:Job@1373] - Job job_1514358672274_0009 completed successfully
2017-12-28 11:34:22,162 [myid:] - INFO  [main:Job@1380] - Counters: 31
        File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=860052
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=913
                HDFS: Number of bytes written=3985558014
                HDFS: Number of read operations=24
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=12
        Job Counters 
                Killed map tasks=1
                Launched map tasks=7
                Other local map tasks=7
                Total time spent by all maps in occupied slots (ms)=1208611
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=1208611
                Total vcore-seconds taken by all map tasks=1208611
                Total megabyte-seconds taken by all map tasks=4331661824
        Map-Reduce Framework
                Map input records=18680041
                Map output records=18680041
                Input split bytes=913
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=4453
                CPU time spent (ms)=180780
                Physical memory (bytes) snapshot=1957969920
                Virtual memory (bytes) snapshot=30116270080
                Total committed heap usage (bytes)=1611661312
        File Input Format Counters 
                Bytes Read=0
        File Output Format Counters 
                Bytes Written=3985558014
2017-12-28 11:34:22,170 [myid:] - INFO  [main:ImportJobBase@184] - Transferred 3.7118 GB in 571.0001 seconds (6.6566 MB/sec)
2017-12-28 11:34:22,174 [myid:] - INFO  [main:ImportJobBase@186] - Retrieved 18680041 records.
2017-12-28 11:34:22,215 [myid:] - INFO  [main:SqlManager@757] - Executing SQL statement: SELECT t.* FROM "rhnpackagefile" AS t LIMIT 1
2017-12-28 11:34:22,245 [myid:] - INFO  [main:HiveImport@194] - Loading uploaded data into Hive
2017-12-28 11:34:28,609 [myid:] - INFO  [Thread-98:LoggingAsyncSink$LoggingThread@85] - 
2017-12-28 11:34:28,609 [myid:] - INFO  [Thread-98:LoggingAsyncSink$LoggingThread@85] - Logging initialized using configuration in jar:file:/u01/hive/lib/hive-common-2.3.2.jar!/hive-log4j2.properties Async: true
2017-12-28 11:34:31,619 [myid:] - INFO  [Thread-98:LoggingAsyncSink$LoggingThread@85] - OK
2017-12-28 11:34:31,622 [myid:] - INFO  [Thread-98:LoggingAsyncSink$LoggingThread@85] - Time taken: 1.666 seconds
2017-12-28 11:34:32,026 [myid:] - INFO  [Thread-98:LoggingAsyncSink$LoggingThread@85] - Loading data to table hivedb.rhnpackagefile
2017-12-28 11:36:14,783 [myid:] - INFO  [Thread-98:LoggingAsyncSink$LoggingThread@85] - OK
2017-12-28 11:36:14,908 [myid:] - INFO  [Thread-98:LoggingAsyncSink$LoggingThread@85] - Time taken: 103.285 seconds
2017-12-28 11:36:15,363 [myid:] - INFO  [main:HiveImport@242] - Hive import complete.
2017-12-28 11:36:15,372 [myid:] - INFO  [main:HiveImport@278] - Export directory is contains the _SUCCESS file only, removing the directory.

2.2 MySQL☞HDFS

[hadoop@hdp01 ~]$ sqoop import --connect jdbc:mysql://192.168.120.92:3306/smsqw --username smsqw -P --table tbDest --columns iMsgID,cDest,tTime,cSMID,iReSend,tLastProcess,cEnCode,tCreateDT,iNum,iResult,iPriority,iPayment,cState,tGpTime --split-by tGpTime --target-dir /user/DataSource/MySQL/tbDest
2017-12-28 14:36:52,550 [myid:] - INFO  [main:Sqoop@92] - Running Sqoop version: 1.4.6
Enter password: 
2017-12-28 14:36:55,496 [myid:] - INFO  [main:MySQLManager@69] - Preparing to use a MySQL streaming resultset.
2017-12-28 14:36:55,497 [myid:] - INFO  [main:CodeGenTool@92] - Beginning code generation
Thu Dec 28 14:36:55 CST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2017-12-28 14:36:56,233 [myid:] - INFO  [main:SqlManager@757] - Executing SQL statement: SELECT t.* FROM `tbDest` AS t LIMIT 1
2017-12-28 14:36:56,253 [myid:] - INFO  [main:SqlManager@757] - Executing SQL statement: SELECT t.* FROM `tbDest` AS t LIMIT 1
2017-12-28 14:36:56,260 [myid:] - INFO  [main:CompilationManager@94] - HADOOP_MAPRED_HOME is /u01/hadoop
Note: /tmp/sqoop-hadoop/compile/4a4024e6b2baa336939a9310f627636a/tbDest.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2017-12-28 14:36:57,637 [myid:] - INFO  [main:CompilationManager@330] - Writing jar file: /tmp/sqoop-hadoop/compile/4a4024e6b2baa336939a9310f627636a/tbDest.jar
2017-12-28 14:36:57,650 [myid:] - WARN  [main:MySQLManager@107] - It looks like you are importing from mysql.
2017-12-28 14:36:57,650 [myid:] - WARN  [main:MySQLManager@108] - This transfer can be faster! Use the --direct
2017-12-28 14:36:57,650 [myid:] - WARN  [main:MySQLManager@109] - option to exercise a MySQL-specific fast path.
2017-12-28 14:36:57,650 [myid:] - INFO  [main:MySQLManager@189] - Setting zero DATETIME behavior to convertToNull (mysql)
2017-12-28 14:36:57,652 [myid:] - INFO  [main:ImportJobBase@235] - Beginning import of tbDest
2017-12-28 14:36:57,653 [myid:] - INFO  [main:Configuration@1019] - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2017-12-28 14:36:57,820 [myid:] - INFO  [main:Configuration@1019] - mapred.jar is deprecated. Instead, use mapreduce.job.jar
2017-12-28 14:36:58,229 [myid:] - INFO  [main:Configuration@1019] - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2017-12-28 14:36:58,581 [myid:] - INFO  [main:TimelineClientImpl@123] - Timeline service address: http://hdp01:8188/ws/v1/timeline/
2017-12-28 14:36:58,770 [myid:] - INFO  [main:AHSProxy@42] - Connecting to Application History server at hdp01.thinkjoy.tt/192.168.120.96:10201
Thu Dec 28 14:37:01 CST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2017-12-28 14:37:01,123 [myid:] - INFO  [main:DBInputFormat@192] - Using read commited transaction isolation
2017-12-28 14:37:01,124 [myid:] - INFO  [main:Configuration@1019] - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2017-12-28 14:37:01,124 [myid:] - INFO  [main:DataDrivenDBInputFormat@147] - BoundingValsQuery: SELECT MIN(`tGpTime`), MAX(`tGpTime`) FROM `tbDest`
2017-12-28 14:37:17,446 [myid:] - INFO  [main:JobSubmitter@396] - number of splits:4
2017-12-28 14:37:17,541 [myid:] - INFO  [main:JobSubmitter@479] - Submitting tokens for job: job_1514358672274_0012
2017-12-28 14:37:17,966 [myid:] - INFO  [main:YarnClientImpl@236] - Submitted application application_1514358672274_0012
2017-12-28 14:37:17,996 [myid:] - INFO  [main:Job@1289] - The url to track the job: http://hdp01:8088/proxy/application_1514358672274_0012/
2017-12-28 14:37:17,996 [myid:] - INFO  [main:Job@1334] - Running job: job_1514358672274_0012
2017-12-28 14:37:26,149 [myid:] - INFO  [main:Job@1355] - Job job_1514358672274_0012 running in uber mode : false
2017-12-28 14:37:26,150 [myid:] - INFO  [main:Job@1362] -  map 0% reduce 0%
2017-12-28 14:39:52,733 [myid:] - INFO  [main:Job@1362] -  map 25% reduce 0%
2017-12-28 14:40:14,978 [myid:] - INFO  [main:Job@1362] -  map 75% reduce 0%
2017-12-28 14:40:43,183 [myid:] - INFO  [main:Job@1362] -  map 100% reduce 0%
2017-12-28 14:40:43,191 [myid:] - INFO  [main:Job@1373] - Job job_1514358672274_0012 completed successfully
2017-12-28 14:40:43,321 [myid:] - INFO  [main:Job@1380] - Counters: 31
        File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=573248
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=609
                HDFS: Number of bytes written=5399155888
                HDFS: Number of read operations=16
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=8
        Job Counters 
                Killed map tasks=2
                Launched map tasks=6
                Other local map tasks=6
                Total time spent by all maps in occupied slots (ms)=724670
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=724670
                Total vcore-seconds taken by all map tasks=724670
                Total megabyte-seconds taken by all map tasks=2597217280
        Map-Reduce Framework
                Map input records=31037531
                Map output records=31037531
                Input split bytes=609
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=3675
                CPU time spent (ms)=588590
                Physical memory (bytes) snapshot=4045189120
                Virtual memory (bytes) snapshot=20141694976
                Total committed heap usage (bytes)=1943535616
        File Input Format Counters 
                Bytes Read=0
        File Output Format Counters 
                Bytes Written=5399155888
2017-12-28 14:40:43,329 [myid:] - INFO  [main:ImportJobBase@184] - Transferred 5.0284 GB in 225.0893 seconds (22.8755 MB/sec)
2017-12-28 14:40:43,335 [myid:] - INFO  [main:ImportJobBase@186] - Retrieved 31037531 records.

2.3 HDFS☞MySQL

[hadoop@hdp01 ~]$ sqoop export --connect jdbc:mysql://192.168.120.92:3306/smsqw?useSSL=false --username smsqw -P --table tbDest2 --export-dir /user/DataSource/MySQL/tbDest
2017-12-28 16:03:18,922 [myid:] - INFO  [main:Sqoop@92] - Running Sqoop version: 1.4.6
Enter password: 
2017-12-28 16:03:21,934 [myid:] - INFO  [main:MySQLManager@69] - Preparing to use a MySQL streaming resultset.
2017-12-28 16:03:21,934 [myid:] - INFO  [main:CodeGenTool@92] - Beginning code generation
2017-12-28 16:03:22,343 [myid:] - INFO  [main:SqlManager@757] - Executing SQL statement: SELECT t.* FROM `tbDest2` AS t LIMIT 1
2017-12-28 16:03:22,365 [myid:] - INFO  [main:SqlManager@757] - Executing SQL statement: SELECT t.* FROM `tbDest2` AS t LIMIT 1
2017-12-28 16:03:22,373 [myid:] - INFO  [main:CompilationManager@94] - HADOOP_MAPRED_HOME is /u01/hadoop
Note: /tmp/sqoop-hadoop/compile/332a6c4b30e942c56cf7f507cdff5761/tbDest2.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2017-12-28 16:03:23,752 [myid:] - INFO  [main:CompilationManager@330] - Writing jar file: /tmp/sqoop-hadoop/compile/332a6c4b30e942c56cf7f507cdff5761/tbDest2.jar
2017-12-28 16:03:23,762 [myid:] - INFO  [main:ExportJobBase@378] - Beginning export of tbDest2
2017-12-28 16:03:23,762 [myid:] - INFO  [main:Configuration@1019] - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2017-12-28 16:03:24,011 [myid:] - INFO  [main:Configuration@1019] - mapred.jar is deprecated. Instead, use mapreduce.job.jar
2017-12-28 16:03:24,738 [myid:] - INFO  [main:Configuration@1019] - mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
2017-12-28 16:03:24,742 [myid:] - INFO  [main:Configuration@1019] - mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
2017-12-28 16:03:24,743 [myid:] - INFO  [main:Configuration@1019] - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2017-12-28 16:03:25,087 [myid:] - INFO  [main:TimelineClientImpl@123] - Timeline service address: http://hdp01:8188/ws/v1/timeline/
2017-12-28 16:03:25,269 [myid:] - INFO  [main:AHSProxy@42] - Connecting to Application History server at hdp01.thinkjoy.tt/192.168.120.96:10201
2017-12-28 16:03:27,400 [myid:] - INFO  [main:FileInputFormat@281] - Total input paths to process : 4
2017-12-28 16:03:27,406 [myid:] - INFO  [main:FileInputFormat@281] - Total input paths to process : 4
2017-12-28 16:03:27,484 [myid:] - INFO  [main:JobSubmitter@396] - number of splits:4
2017-12-28 16:03:27,493 [myid:] - INFO  [main:Configuration@1019] - mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
2017-12-28 16:03:27,493 [myid:] - INFO  [main:Configuration@1019] - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2017-12-28 16:03:27,577 [myid:] - INFO  [main:JobSubmitter@479] - Submitting tokens for job: job_1514358672274_0020
2017-12-28 16:03:28,062 [myid:] - INFO  [main:YarnClientImpl@236] - Submitted application application_1514358672274_0020
2017-12-28 16:03:28,091 [myid:] - INFO  [main:Job@1289] - The url to track the job: http://hdp01:8088/proxy/application_1514358672274_0020/
2017-12-28 16:03:28,092 [myid:] - INFO  [main:Job@1334] - Running job: job_1514358672274_0020
2017-12-28 16:17:18,663 [myid:] - INFO  [main:Job@1355] - Job job_1514358672274_0020 running in uber mode : false
2017-12-28 16:17:18,665 [myid:] - INFO  [main:Job@1362] -  map 0% reduce 0%
2017-12-28 16:17:34,148 [myid:] - INFO  [main:Job@1362] -  map 1% reduce 0%
2017-12-28 16:17:43,200 [myid:] - INFO  [main:Job@1362] -  map 2% reduce 0%
2017-12-28 16:17:55,269 [myid:] - INFO  [main:Job@1362] -  map 3% reduce 0%
......
2017-12-28 16:40:15,427 [myid:] - INFO  [main:Job@1362] -  map 100% reduce 0%
2017-12-28 16:40:32,491 [myid:] - INFO  [main:Job@1373] - Job job_1514358672274_0020 completed successfully
2017-12-28 16:40:32,659 [myid:] - INFO  [main:Job@1380] - Counters: 31
        File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=571960
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=5401517442
                HDFS: Number of bytes written=0
                HDFS: Number of read operations=70
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=0
        Job Counters 
                Launched map tasks=4
                Other local map tasks=1
                Rack-local map tasks=3
                Total time spent by all maps in occupied slots (ms)=4931826
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=4931826
                Total vcore-seconds taken by all map tasks=4931826
                Total megabyte-seconds taken by all map tasks=17675664384
        Map-Reduce Framework
                Map input records=31037531
                Map output records=31037531
                Input split bytes=2192
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=21815
                CPU time spent (ms)=1522470
                Physical memory (bytes) snapshot=3453595648
                Virtual memory (bytes) snapshot=20112125952
                Total committed heap usage (bytes)=477102080
        File Input Format Counters 
                Bytes Read=0
        File Output Format Counters 
                Bytes Written=0
2017-12-28 16:40:32,667 [myid:] - INFO  [main:ExportJobBase@301] - Transferred 5.0306 GB in 2,227.9141 seconds (2.3122 MB/sec)
2017-12-28 16:40:32,671 [myid:] - INFO  [main:ExportJobBase@303] - Exported 31037531 records.

另附import和export常用参数说明表：
Using Sqoop 1.4.6 With Hadoop 2.7.4
Using Sqoop 1.4.6 With Hadoop 2.7.4

weixin_33788244
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Hadoop学习笔记（九）—— Sqoop安装配置
JCManZZ的博客
05-14 675
文章目录前言第一步 安装Sqoop第二步 修改必要的配置文件修改sqoop-env.sh配置文件配置sqoop系统环境变量效果展示 前言 Sqoop是一个数据迁移工具，通过使用Sqoop可以将数据从关系数据库导入到Hadoop文件系统中，反之亦可。Sqoop主要分为Sqoop1和Sqoop2两个版本，其中Sqoop1部署方便且结构简单，适合简单的数据迁移工作。这里只是用于学习需要，所以只展示部署Sqoop1的步骤。 第一步 安装Sqoop 这里还是使用rz命令上传下载好的sqoop-1.4.6.bin__h
基于Hadoop-2.7.7的sqoop-1.4.6安装文档
鲨鱼胃的博客
03-26 578
1 准备安装包 2 解压安装包 上传安装包 解压安装包 [yao@master ~]$ tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz 3 复制sqoop-env.sh文件 把解压后的安装目录下的conf目录中的sqoop-env-templete.sh 复制到sqoop-env.sh [yao@master sqoop-1....
【大数据开发运维解决方案】sqoop增量导入oracle/mysql数据到hive时时间字段为null处理
热门推荐
赵延东的一亩三分地
03-27 16万+
前面几篇文章详细介绍了sqoop全量增量导入数据到hive，比如文章（sqoop避免输入密码自动增量job脚本介绍）里面的案例和给出的其他相关文章的连接，笔者使用sqoop导入oracle数据到hive是为了做增量cube用，但是在前面的某篇文章Sqoop全量同步mysql/Oracle数据到hive中大家可以看到我导入的数据如果有时间字段的话我都是在hive指定成了string类型，虽然这样可以处理掉时间字段在hive为空的问题，但是在kylin创建增量cube时需要指定一个时间字段来做增量，所以上面那种
sqoop java.lang.NullPointerException 和 No columns to generate for ClassWriter
小陌成长之路
07-31 1万+
running Sqoop: java.lang.NullPointerException SQL command not properly ended java.io.IOException: No columns to generate for ClassWriter
sqoop query时单双引号选用以及$CONDITION使用的探究
qq_34341930的博客
11-03 1257
这段时间碰见了一个奇怪的sqoop导入问题。我的业务目标是想将postgresql库里的某张表内的数据导入到hive里，而且在导入的时候需要做一步查询。但在导入的时候，围绕着"$CONDITION"这个参数，会有不同的运行结果，有的报错，有的查询结果为空，我们下面来看看： 在pgsql里的查询语句为： # postgresql 查询语句 select id,check_code,lastupdatetime,check_num,report_name,json_object_keys(reportin
sqoop-1.4.6-hadoop-2.6最小资源包
09-14
这个"sqoop-1.4.6-hadoop-2.6 最小资源包"是针对 Sqoop 1.4.6 版本，专为运行在 Hadoop 2.6 环境优化的精简版安装包。 Sqoop 的主要功能包括： 1. **数据导入**：通过 SQL 查询语句， Sqoop 可以选择性地从数据库...
sqoop-1.4.6.bin-hadoop-2.0.4-alpha版本的压缩包，直接下载到本地，解压后即可使用
最新发布
07-02
Sqoop(发音：skup)是一款开源的工具，主要用于在Hadoop(Hive)与传统的数据库(mysql、postgresql...)间进行数据的传递，可以将一个关系型数据库（例如 ： MySQL ,Oracle ,Postgres等）中的数据导进到Hadoop的HDFS中，...
sqoop-1.4.6-cdh5.14.0
08-04
在这个版本 "sqoop-1.4.6-cdh5.14.0" 中，它已经集成到 Cloudera Distribution Including Hadoop (CDH) 的 5.14.0 版本中，提供了更稳定和高效的数据导入导出功能。 一、Sqoop 的核心功能 1. 数据导入：Sqoop 可以...
sqoop-1.4.6.bin__hadoop-2.0.4-alpha.zip
09-22
这个压缩包 "sqoop-1.4.6.bin__hadoop-2.0.4-alpha.zip" 包含的是Sqoop 1.4.6版本，针对Hadoop 2.0.4-alpha版本优化的二进制发行版。 **Sqoop 的核心功能：** 1. **数据导入**：Sqoop 提供了命令行接口，可以将结构...
sqoop-1.4.6.bin__hadoop-2.0.4-alpha安装包
03-05
Sqoop是一个用于在Hadoop和关系数据库或大型机之间传输数据的工具。您可以使用Sqoop将关系数据库管理系统（RDBMS）中的数据导入Hadoop分布式文件系统（HDFS），转换Hadoop MapReduce中的数据，然后将数据导出回RDBMS...
sqoop测试连接oracle,sqoop测试oracle数据库的连接使用
weixin_29243279的博客
04-09 1290
测试oracle数据库的连接使用①连接oracle数据库，列出所有的数据库[hadoop@eb179 sqoop]$sqoop list-databases--connect jdbc 10.1.69.173:1521:ORCLBI --userna[hadoop@eb179 sqoop]$sqoop list-databases--connect jdbc 10.1.69.173:1521:OR...
sqoop配置以及使用过程小结
在路上的学习者
10-20 7595
使用sqoop用来将关系型数据库里的内容与HDFS上的内容互转。本文用sqoop测试2个内容： （1）将pg上数据移动到hdfs上； （2）将hdfs上数据移动到pg上； hadoop版本：2.7.1 sqoop版本：sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz pg版本：9.4.1
使用Sqoop进行数据的导入
实践求真知
12-04 7210
一 下载地址 http://archive.apache.org/dist/sqoop/1.4.4/ [root@localhost bin]# wget http://archive.apache.org/dist/sqoop/1.4.4/sqoop-1.4.4.bin__hadoop-0.20.tar.gz 二 解压sqoop [root@localhost bin]# tar
Sqoop初次正式使用所遇疑难杂症疗方
weixin_46001623的博客
03-18 1769
SQOOP 疑难杂症疗方前言测试sqoop 能否连的上数据库这时第一个错误出现了疗方现在我们可以尝试抽数了这时我们可能会遇到第二个问题疗方成功后 我便用一个比较大点的表来进行测试所以第三个问题来了疗方 前言 因为在公司后台架构组件中已经安装好了Sqoop 所以我这里直接使用即可。 可以使用此命令来查看是否有安装sqoop 和其版本 sqoop version [hdfs@AnalytoolNode ~]$ sqoop version Warning: /opt/cloudera/parcels/CDH
hdfs导入mysql出错_我用sqoop从oracle导入数据到hdfs时，总是报 表或视图不存在错误，求解答......
weixin_29698641的博客
02-15 151
表在oracle，XE数据库实例中已经建好，并且命令中的名字也没有错具体的信息[root@hadoop03bin]#./sqoopimport--connectjdbc:oracle:thin:@192.168.8.100:1521:xe--usernamehadoop--pa...表在oracle，XE数据库实例中已经建好，并且命令中的名字也没有错具体的信息[root@hadoop03 bin]...
sqoop的基本语法详解及可能遇到的错误
01-15 1万+
1 sqoop介绍 Apache Sqoop是专为Apache Hadoop和结构化数据存储如关系数据库之间的数据转换工具的有效工具。你可以使用Sqoop从外部结构化数据存储的数据导入到Hadoop分布式文件系统或相关系统如Hive和HBase。相反，Sqoop可以用来从Hadoop的数据提取和导出到外部结构化数据存储如关系数据库和企业数据仓库。 Sqoop专为大数据批量传输设计，能够分割数...
Linux搭建sqoop数据迁移工具及简单的数据迁移测试
汉南最後の読書人
07-11 2386
1.准备环境及工具 JDK 1.8 已搭建 Hadoop 3.1.1 已搭建 zookeeper 3.5.5 下载 hbase 2.2.0 下载 hive 3.1.1 已搭建 sqoop-1.4.7 下载 ojdbc8(该版本的驱动支持jdk1.8) 下载 2.搭建zookeeper 3.5.5单机模式 2.1 解压zookeeper tar -zxvf apach...
Sqoop1.4.6使用数据导入导出
渣渣
07-24 1379
sqoop是一款用于结合关系型数据库和hdfs(hive/hbase)的数据库之间数据相互传递的工具，可以将mysql/oracle等数据库中存在的表格通过sqoop来传递给hive或着是hbase，同样也可以将hive或者是hbase中的表格传递给sqoop中，非常好用。 目前主要有两种版本的sqoop，分别是sqoop-1.4.*和sqoop-1.99.*。其中sqoop-1.4.*代表的是
oracle sqoop 导出_Sqoop从Oracle导出数据出错：The Network
weixin_42368683的博客
01-13 256
Sqoop从Oracle导出数据出错：The Network发布时间：2020-04-09 12:24:27来源：51CTO阅读：211作者：dwf07223Sqoop执行命令：./sqoop import --connect jdbc:oracle:thin:@10.112.101.251:1621:crmadbmr--username bass_etl --password 75!n!u...
Sqoop-1.4.6安装配置及Mysql->HDFS->Hive数据导入（基于Hadoop2.7.3）
小太阳~
07-15 5487
写在前面 本博客基于的环境如下： OS版本： Ubuntu16.04 Hadoop版本： Hadoop-2.7.3 Hbase版本： Hbase-1.3.1 Hive版本： Hive-2.1.1 Pig版本： Pig-0.16.0 Zookeeper版本： Zookeeper-3.4.9 Sqoop版本： Sqoop-1.4.6 一、Sqoop安装 1、下载 下载地址： ht...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值