sqoop:mysql import to hdfs

最新推荐文章于 2022-12-02 15:16:48 发布
最新推荐文章于 2022-12-02 15:16:48 发布
阅读量304 收藏
点赞数
分类专栏： sqoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_29416019/article/details/83893534
版权

sqoop import
–connect jdbc:mysql://localhost:3306/chen
–username root
–password 123
–table test

报错：
Exception in thread “main” java.lang.NoClassDefFoundError: org/json/JSONObject
at org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:42)

解决方案：

下载 java-json.jar

导出的位置： hdfs dfs -ls /user/hadoop

导出日志：

18/10/30 09:46:59 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
18/10/30 09:46:59 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/10/30 09:46:59 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/10/30 09:46:59 INFO tool.CodeGenTool: Beginning code generation
18/10/30 09:46:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM test AS t LIMIT 1
18/10/30 09:46:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM test AS t LIMIT 1
18/10/30 09:46:59 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/app/hadoop
Note: /tmp/sqoop-hadoop/compile/c09c40ce6aefc16447052c26316ab4af/test.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/10/30 09:47:01 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/c09c40ce6aefc16447052c26316ab4af/test.jar
18/10/30 09:47:01 WARN manager.MySQLManager: It looks like you are importing from mysql.
18/10/30 09:47:01 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
18/10/30 09:47:01 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
18/10/30 09:47:01 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
18/10/30 09:47:01 INFO mapreduce.ImportJobBase: Beginning import of test
18/10/30 09:47:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
18/10/30 09:47:02 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/10/30 09:47:02 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/10/30 09:47:02 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/10/30 09:47:04 INFO db.DBInputFormat: Using read commited transaction isolation
18/10/30 09:47:04 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(user), MAX(user) FROM test
18/10/30 09:47:04 WARN db.TextSplitter: Generating splits for a textual index column.
18/10/30 09:47:04 WARN db.TextSplitter: If your database sorts in a case-insensitive order, this may result in a partial import or duplicate records.
18/10/30 09:47:04 WARN db.TextSplitter: You are strongly encouraged to choose an integral split column.
18/10/30 09:47:04 INFO mapreduce.JobSubmitter: number of splits:4
18/10/30 09:47:04 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1540200454240_0014
18/10/30 09:47:04 INFO impl.YarnClientImpl: Submitted application application_1540200454240_0014
18/10/30 09:47:04 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1540200454240_0014/
18/10/30 09:47:04 INFO mapreduce.Job: Running job: job_1540200454240_0014
18/10/30 09:47:12 INFO mapreduce.Job: Job job_1540200454240_0014 running in uber mode : false
18/10/30 09:47:12 INFO mapreduce.Job: map 0% reduce 0%
18/10/30 09:47:22 INFO mapreduce.Job: map 25% reduce 0%
18/10/30 09:47:24 INFO mapreduce.Job: map 50% reduce 0%
18/10/30 09:47:25 INFO mapreduce.Job: map 75% reduce 0%
18/10/30 09:47:26 INFO mapreduce.Job: map 100% reduce 0%
18/10/30 09:47:26 INFO mapreduce.Job: Job job_1540200454240_0014 completed successfully
18/10/30 09:47:26 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=550136
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=493
HDFS: Number of bytes written=51
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=35952
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=35952
Total vcore-seconds taken by all map tasks=35952
Total megabyte-seconds taken by all map tasks=36814848
Map-Reduce Framework
Map input records=4
Map output records=4
Input split bytes=493
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=184
CPU time spent (ms)=3950
Physical memory (bytes) snapshot=703848448
Virtual memory (bytes) snapshot=6232666112
Total committed heap usage (bytes)=457179136
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=51
18/10/30 09:47:26 INFO mapreduce.ImportJobBase: Transferred 51 bytes in 23.7328 seconds (2.1489 bytes/sec)
18/10/30 09:47:26 INFO mapreduce.ImportJobBase: Retrieved 4 records

sqoop import
–connect jdbc:mysql://localhost:3306/chen
–username root
–password 123
–table test
–mapreduce-job-name FromMySQLToHDFS
–delete-target-dir
-m 1

常用参数：

–mapreduce-job-name 改MR名字
-m 默认是4 能忍受就行 不要太大
–columns 需要导出的列
–target-dir 目标位置
–null-non-string ‘0’ \ 非字符串空值转换
–null-string ‘’ \ 字符串空值转换
–where ‘SAL>2000’ where

示例：

sqoop import
–connect jdbc:mysql://localhost:3306/chen
–username root
–password 123
–table chen
–mapreduce-job-name FromMySQLToHDFS
–delete-target-dir
-m 1
–null-non-string ‘0’
–null-string ‘0’

sqoop import
–connect jdbc:mysql://localhost:3306/chen
–username root
–password 123
–table chen
–mapreduce-job-name FromMySQLToHDFS
–delete-target-dir
-m 1
–null-non-string ‘0’
–null-string ‘0’
–where “user=‘chen’”

sqoop import
–connect jdbc:mysql://localhost:3306/chen
–username root
–password 123
–mapreduce-job-name FromMySQLToHDFS
-m 1
–delete-target-dir
–target-dir EMP_COLUMN_QUERY
–fields-terminated-by ‘\t’
–null-non-string ‘0’
–null-string ‘’
-e ‘select * from chen where host is not null and $CONDITIONS’

或者 “SELECT * FROM x WHERE a=‘foo’ AND $CONDITIONS”

sqoop import
–connect jdbc:mysql://localhost:3306/chen
–username root
–password 123
–table d5_emp
–delete-target-dir
–fields-terminated-by ‘\t’
–target-dir EMP_COLUMN_columns
–columns ‘EMPNO,ENAME,JOB,SAL,COMM’
-m 1

风继续吹-小海
关注
专栏目录
Sqoop01：Sqoop安装部署以及快速使用：从MySQL导入HDFS、从HDFS导出MySQL
weixin_40612128的博客
03-10 3805
一、Sqoop的版本 Sqoop目前有两大版本，Sqoop1和Sqoop2，这两个版本都是一直在维护者的，所以使用哪个版本都可以。 这两个版本我都用过，还是感觉Sqoop1用起来比较方便，使用Sqoop1的时候可以将具体的命令全部都写到脚本中，这样看起来是比较清晰的，但是有一个弊端，就是在操作MySQL的时候，MySQL数据库的用户名和密码会明文暴露在这些脚本中，不过一般也没有什么问题，因为在访问生产环境下的MySQL的时候，是需要申请权限的，就算你知道了MySQL的用户名和密码，但是你压根无法访问MySQ
使用Sqoop实现Mysql到HDFS的数据导入
QYHuiiQ
05-11 3316
查看当前mysql下有哪些库 [root@hadoop01 ~]# sqoop list-databases --connect jdbc:mysql://hadoop03:3306 --username root --password 123456; 查看某个库下的所有表 [root@hadoop01 ~]# sqoop list-tables --connect jdbc:mysql://hadoop03:3306/hive --username root --password 12345.
sqoop导入数据到hdfs路径
11-08
sqoop导入数据到hdfs，所有相关的操作命令均在更改相关地址就行。
sqoop:mysql to hdfs
Mike9955的博客
12-07 175
将mysql联合查询的结果导入到hdfs上 sqoop import --connect jdbc:mysql://ip/数据库 --username root --password 密码 --query "select p.bidNo as bidNo,p.interestTime as interestTime,p.status as status,p.planStatus a...
sqoop(2):import之mysqlToHDFS
小蚯蚓的博客
11-11 367
一、实现功能 将mysql数据库中数据导入到hdfs上。 二、方法 1.mysql中新建测试表 create database if not exists sqoop; use sqoop; //创建测试表 create table tohdfs( id int primary key not null, name varchar(20) not null ); //插入测试数据 ...
mysql to hdfs_sqoop:mysql to hdfs
weixin_28923083的博客
01-20 80
sqoop import --connect jdbc:mysql://ip/数据库 --username root --password 密码 --query "select p.bidNo as bidNo,p.interestTime as interestTime,p.status as status,p.planStatus as planStatus,i.yield as yield,...
Spark Mysql to hdfs
u012711151的博客
08-18 272
import org.apache.hadoop.fs.Path import org.apache.spark.sql.SQLContext import org.apache.spark.{SparkConf, SparkContext}     object MysqlToHdfs {    def main(args: Array[String]) {        v...
Sqoop Export HDFS
pigziprogrammer的专栏
07-02 143
Sqoop Export 应用场景——直接导出 直接导出 sqoop export \ --connect 'jdbc:mysql://xxx.xxx.xxx.xxx/test?useUnicode=true&characterEncodi g=utf-8' \ --username root \ --password-file /user/hadoop/.password ...
Sqoop导入mysql数据到HDFS
蜗牛往上爬
08-11 2214
Sqoop导入mysql数据到HDFS 打算把mysql中爬虫爬到的数据导入到HDFS中供后续学习kylin以及研究使用，以前导数据经常会自己写一段java脚本去导入，这次打算用sqoop试一试效果 首先看了下mysql的数据量，不到900万条数据，不多不少 从官网下载好sqoop进行安装 http://sqoop.apache.org/docs/1.99.7/admin/Inst...
sqoop: mysql导入数据到hdfs, hive, hbase
yao09605的博客
01-19 375
mysql 导入 hdfs #启动hadoop $ start-dfs.sh $ start-yarn.sh $ jps 2706 NameNode 3334 ResourceManager 3495 NodeManager 3112 SecondaryNameNode 3848 Jps 2873 DataNode # sqoop energydata是数据库名，average_price...
Sqoop数据导入 第2关：Mysql导入数据至HDFS上
最新发布
qq_61604164的博客
12-02 2742
第2关：Mysql导入数据至HDFS上 Sqoop数据导入
Talend学习笔记2——mysql文件导入到HDFS
06-01
Talend学习笔记2——mysql文件导入到HDFS
Sqoop从Mysql导入HDFS、HIVE
斑马的地盘
01-14 479
Sqoop全量导入mysql到HDFS bin/sqoop import \ 数据库名 --connect jdbc:mysql://node-1:3306/userdb \ --username root \ --password hadoop \ --delete-targer-dir \ -- targer-dir /sqoopresult \ 指定数据在HDFS中的路径 --fields-terminated-by '\t' \ 指定字段分隔符，默认为"," --split-by id 当MT大于一
mysql to hdfs_Spark Mysql to hdfs
weixin_39738416的博客
02-01 79
import org.apache.hadoop.fs.Pathimport org.apache.spark.sql.SQLContextimport org.apache.spark.{SparkConf, SparkContext}object MysqlToHdfs {def main(args: Array[String]) {val conf = new SparkConf().set...
sqoop导入脚本——mysql_to_hdfs
LZX的博客
11-11 429
#!/bin/bash #操作命令格式 #sh mysql_to_hdfs.sh first 开始日期 结束日期（默认开始时间为前一天，结束时间为当天） sqoop=/usr/hdp/3.1.4.0-315/sqoop/bin/sqoop db=SAJT_DZFP_PT do_date=`date -d '-1 day' +%F` endDay=`date +%F` if [[ -n "$2" ]]; then do_date=$2 fi if [[ -n "$3" ]]; then
sqoop离线数据迁移 --- MYSQL TO HDFS
小小码农
03-26 257
mysql导入数据到hdfs （sqoop） # 准备： [root@leader sqoop-1.4.7.bin__hadoop-2.6.0]# hdfs dfs -mkdir -p /sqoop/mysql/ [root@leader sqoop-1.4.7.bin__hadoop-2.6.0]# hdfs dfs -ls -R /sqoop drwxr-xr-x - root supergroup 0 2022-03-26 02:40 /sqoop/mysql # --n
数据仓库之mysql业务数据上传到hdfs
guohuishan的博客
08-20 666
业务数据就是业务流程中产生的登陆、订单、用户、商品、支付等相关的数据，通常存储在DB中，包括mysql、oracle等 mysql版本：5.6.24 sqoop版本：sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz hadoop版本：3.2.1 编写脚本mysql_to_hdfs.sh #! /bin/bash sqoop=/opt/module/sqoop/bin/sqoop do_date=`date -d '-1 day' +%F` if [[ -n "$2"
mysql与hdfs的数据迁移（sqoop）
Above the cloud
08-14 616
文章目录1、查看mysql数据2、查看mysql表3、mysql表导入到hdfs4、查看hdfs5、清空dept表6、数据从hdfs导出到MySQL表7、查询dept表8、Sqoop Import 增量导入到 HDFS 打开一个 dektop 终端，不要打开多个终端，所用数据看上一篇 1、查看mysql数据 hadoop@ddai-desktop:~$ sqoop list-databases --connect jdbc:mysql://ddai-master:3306/ --username sqoo
[原创] MysqlsqoopHDFS 数据交换实验
congnangwu6519的博客
12-18 163
感谢@吴飚飚哥的细心指导，向飚哥精益求精的精神学习 SQOOP：Apache基金会下一个开源产品，Hadoop家族的一个产品，关系型数据库与HDFS文件系统之间进行数据交换，数据迁移的一个工具。 ...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值