CDH5.9.0 +sqoop2 [1.99.5]数据传输

最新推荐文章于 2022-05-14 16:19:30 发布
最新推荐文章于 2022-05-14 16:19:30 发布
阅读量4.9k 收藏 1
点赞数
分类专栏： sqoop2

先添加sqoop2
这里写图片描述
然后选择sqoop2，然后就没有什么了完成后启动。
去/var/lib/下面添加mysql驱动

在执行

cd /opt/cloudera/parcels/CDH/lib/hadoop/client
sudo ln -s ../../hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar 
sudo ln -s ../../hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar 
`
 ``

之后就可以了

进到安装的机器上
执行sqoop2
进去shell、

sqoop:000> set server –host your.host.com –port 12000 –webapp sqoop

sqoop:000> show version –all
client version:
Sqoop 2.0.0-SNAPSHOT source revision 418c5f637c3f09b94ea7fc3b0a4610831373a25f
Compiled by vbasavaraj on Mon Nov 3 08:18:21 PST 2014
server version:
Sqoop 2.0.0-SNAPSHOT source revision 418c5f637c3f09b94ea7fc3b0a4610831373a25f
Compiled by vbasavaraj on Mon Nov 3 08:18:21 PST 2014
API versions:
[v1]
sqoop:000> help
For information about Sqoop, visit: http://sqoop.apache.org/

Available commands:
exit (\x ) Exit the shell
history (\H ) Display, manage and recall edit-line history
help (\h ) Display this help message
set (\st ) Configure various client options and settings
show (\sh ) Display various objects and configuration options
create (\cr ) Create new object in Sqoop repository
delete (\d ) Delete existing object in Sqoop repository
update (\up ) Update objects in Sqoop repository
clone (\cl ) Create new object based on existing one
start (\sta) Start job
stop (\stp) Stop job
status (\stu) Display status of a job
enable (\en ) Enable object in Sqoop repository
disable (\di ) Disable object in Sqoop repository

sqoop:000> show connector
+—-+————————+—————-+——————————————————+———————-+
| Id | Name | Version | Class | Supported Directions |
+—-+————————+—————-+——————————————————+———————-+
| 1 | hdfs-connector | 2.0.0-SNAPSHOT | org.apache.sqoop.connector.hdfs.HdfsConnector | FROM/TO |
| 2 | generic-jdbc-connector | 2.0.0-SNAPSHOT | org.apache.sqoop.connector.jdbc.GenericJdbcConnector | FROM/TO |
+—-+————————+—————-+——————————————————+———————-+

还有show link ,show job
创建link，job。的去官网这里不说了

再说下定时启动job
写一个java api

这个代码显示不一样。见谅吧

package com.lenovo.analyze.sqoopmtm;

import org.apache.sqoop.client.SqoopClient;
import org.apache.sqoop.model.MConnector;
import org.apache.sqoop.model.MDriverConfig;
import org.apache.sqoop.model.MFromConfig;
import org.apache.sqoop.model.MJob;
import org.apache.sqoop.model.MLink;
import org.apache.sqoop.model.MLinkConfig;
import org.apache.sqoop.model.MSubmission;
import org.apache.sqoop.model.MToConfig;
import org.apache.sqoop.submission.counter.Counter;
import org.apache.sqoop.submission.counter.CounterGroup;
import org.apache.sqoop.submission.counter.Counters;
import org.apache.sqoop.validation.Status;
public class MysqlToHDFS {

public static void main(String[] args) {
sqoopTransfer();
}
public static void sqoopTransfer() {
//初始化
String url =”http://ip:12000/sqoop/”;
SqoopClient client = new SqoopClient(url);
client.setServerUrl(url);

java.util.Random r=new java.util.Random();


int num=r.nextInt();
    //创建一个源链接 JDBC
    long fromConnectorId = 1;
    MLink fromLink =client.createLink(fromConnectorId);
    fromLink.setName("JDBC"+num);

// fromLink.setCreationUser(“liuwei”);
MLinkConfig fromLinkConfig =fromLink.getConnectorLinkConfig();
fromLinkConfig.getStringInput(“linkConfig.connectionString”).setValue(“jdbc:mysql://ip:3306/db_warehouse”);
fromLinkConfig.getStringInput(“linkConfig.jdbcDriver”).setValue(“com.mysql.jdbc.Driver”);
fromLinkConfig.getStringInput(“linkConfig.username”).setValue(“yonghu”);
fromLinkConfig.getStringInput(“linkConfig.password”).setValue(“888!@#”);
Status fromStatus =client.saveLink(fromLink);
System.out.println(fromStatus);
if(fromStatus.canProceed()) {
System.out.println(“创建JDBC Link成功，ID为:” + fromLink.getPersistenceId());
} else {
System.out.println(“创建JDBC Link失败”);
}

    //创建一个目的地链接HDFS
    long toConnectorId = 3;
    MLink toLink =client.createLink(toConnectorId);
    toLink.setName("HDFS"+num);
    toLink.setCreationUser("sqoop2");
    MLinkConfig toLinkConfig =toLink.getConnectorLinkConfig();
    System.out.println(toLinkConfig);
    toLinkConfig.getStringInput("linkConfig.uri").setValue("hdfs://ip:8020/");
    Status toStatus =client.saveLink(toLink);
    if(toStatus.canProceed()) {
    System.out.println("创建HDFS Link成功，ID为:" + toLink.getPersistenceId());
    } else {
    System.out.println("创建HDFS Link失败");
    }

    //创建一个任务
    long fromLinkId =fromLink.getPersistenceId();
    long toLinkId =toLink.getPersistenceId();
    MJob job = client.createJob(fromLinkId,toLinkId);
    job.setName("MySQL to HDFSjob"+num);
    job.setCreationUser("hadoop");
    //设置源链接任务配置信息
    MFromConfig fromJobConfig = job.getFromJobConfig();
    System.out.println("****"+fromJobConfig);
    fromJobConfig.getStringInput("fromJobConfig.schemaName").setValue("db_warehouse");
    fromJobConfig.getStringInput("fromJobConfig.tableName").setValue("T_TEST");
    fromJobConfig.getStringInput("fromJobConfig.partitionColumn").setValue("id");
   fromJobConfig.getStringInput("fromJobConfig.columns").setValue("id,age");
    MToConfig toJobConfig =job.getToJobConfig();
    System.out.println("===="+toJobConfig);

// toJobConfig.getEnumInput(“toJobConfig.outputFormat”).setValue(“TEXT_FILE”);
toJobConfig.getEnumInput(“toJobConfig.outputFormat”).setValue(“TEXT_FILE”);
toJobConfig.getEnumInput(“toJobConfig.compression”).setValue(“NONE”);
toJobConfig.getStringInput(“toJobConfig.outputDirectory”).setValue(“/user/sqoop2/”);
MDriverConfig driverConfig =job.getDriverConfig();
driverConfig.getIntegerInput(“throttlingConfig.numExtractors”).setValue(3);

    Status status = client.saveJob(job);
    System.out.println("===="+status);
    if(status.canProceed()) {
    System.out.println("JOB创建成功，ID为:"+ job.getPersistenceId());
    } else {
    System.out.println("JOB创建失败。");
    }



    //启动任务
    long jobId = job.getPersistenceId();
       System.out.println(jobId);

// long jobId = 4;
MSubmission submission =client.startJob(jobId);
System.out.println(“JOB提交状态为 : ” + submission.getStatus());
while(submission.getStatus().isRunning()&& submission.getProgress() != -1) {
System.out.println(“进度 : ” + String.format(“%.2f%%”, submission.getProgress() * 100));
//三秒报告一次进度
try {
Thread.sleep(3000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}

    System.out.println("JOB执行结束... ...");
    System.out.println("Hadoop任务ID为:" + submission.getExternalJobId());
    Counters counters =submission.getCounters();
    if(counters != null) {
    System.out.println("计数器:");
    for(CounterGroup group : counters) {
    System.out.print("\t");
    System.out.println(group.getName());
    for(Counter counter : group) {
    System.out.print("\t\t");
    System.out.print(counter.getName());
    System.out.print(": ");
    System.out.println(counter.getValue());
    }
    }
    }
    if(submission.getError().getErrorSummary() != null) {
    System.out.println("JOB执行异常，异常信息为 : " +submission.getError());
    }
    System.out.println("HDFS通过sqoop传输数据到MySQL统计执行完毕");

    }
    }

package com.lenovo.analyze.sqoopmtm;

import org.apache.sqoop.client.SqoopClient;
import org.apache.sqoop.model.MDriverConfig;
import org.apache.sqoop.model.MFromConfig;
import org.apache.sqoop.model.MJob;
import org.apache.sqoop.model.MLink;
import org.apache.sqoop.model.MLinkConfig;
import org.apache.sqoop.model.MSubmission;
import org.apache.sqoop.model.MToConfig;
import org.apache.sqoop.submission.counter.Counter;
import org.apache.sqoop.submission.counter.CounterGroup;
import org.apache.sqoop.submission.counter.Counters;
import org.apache.sqoop.validation.Status;
public class HDFSToMysql {
public static void main(String[] args) {
sqoopTransfer();
}
public static void sqoopTransfer() {
//初始化
String url =”http://ip:12000/sqoop/”;
SqoopClient client = new SqoopClient(url);
int num=205;

    //创建一个源链接 HDFS
    long fromConnectorId = 3;
    MLink fromLink =client.createLink(fromConnectorId);
    fromLink.setName("HDFSconnector"+num);
    fromLink.setCreationUser("admln");
    MLinkConfig fromLinkConfig =fromLink.getConnectorLinkConfig();
    fromLinkConfig.getStringInput("linkConfig.uri").setValue("hdfs://ip-hadoop02:8020/");
    Status fromStatus =client.saveLink(fromLink);
    if(fromStatus.canProceed()) {
    System.out.println("创建HDFS Link成功，ID为:" + fromLink.getPersistenceId());
    } else {
    System.out.println("创建HDFS Link失败");
    }
    //创建一个目的地链接 JDBC
    long toConnectorId = 1;
    MLink toLink = client.createLink(toConnectorId);
    toLink.setName("JDBCconnector"+num);
    toLink.setCreationUser("admln");
    MLinkConfig toLinkConfig =toLink.getConnectorLinkConfig();
    toLinkConfig.getStringInput("linkConfig.connectionString").setValue("jdbc:mysql://ip/db_warehouse");
    toLinkConfig.getStringInput("linkConfig.jdbcDriver").setValue("com.mysql.jdbc.Driver");
    toLinkConfig.getStringInput("linkConfig.username").setValue("yonghu");
    toLinkConfig.getStringInput("linkConfig.password").setValue("9999!@#");
    Status toStatus =client.saveLink(toLink);
    if(toStatus.canProceed()) {
    System.out.println("创建JDBC Link成功，ID为:" + toLink.getPersistenceId());
    } else {
    System.out.println("创建JDBC Link失败");
    }

    //创建一个任务
    long fromLinkId =fromLink.getPersistenceId();
    long toLinkId =toLink.getPersistenceId();
    MJob job = client.createJob(fromLinkId,toLinkId);
    job.setName("HDFS to MySQLjob"+num);
    job.setCreationUser("sqoop2");
    //设置源链接任务配置信息
    MFromConfig fromJobConfig =job.getFromJobConfig();
    fromJobConfig.getStringInput("fromJobConfig.inputDirectory").setValue("/user/sqoop2");

    //创建目的地链接任务配置信息
    MToConfig toJobConfig =job.getToJobConfig();
    toJobConfig.getStringInput("toJobConfig.schemaName").setValue("db_warehouse");
    toJobConfig.getStringInput("toJobConfig.tableName").setValue("T_TEST_copy");
    //toJobConfig.getStringInput("fromJobConfig.partitionColumn").setValue("id");
    // set the driver config values

// MDriverConfig driverConfig =job.getDriverConfig();
// driverConfig.getIntegerInput(“throttlingConfig.numExtractors”).setValue(3);//这句还没弄明白
Status status = client.saveJob(job);
if(status.canProceed()) {
System.out.println(“JOB创建成功，ID为:”+ job.getPersistenceId());
} else {
System.out.println(“JOB创建失败。”);
}

    //启动任务
    long jobId = job.getPersistenceId();
    MSubmission submission =client.startJob(jobId);
    System.out.println("JOB提交状态为 : " + submission.getStatus());
    while(submission.getStatus().isRunning() &&submission.getProgress() != -1) {
    System.out.println("进度 : " + String.format("%.2f%%", submission.getProgress() * 100));
    //三秒报告一次进度
    try {
    Thread.sleep(3000);
    } catch (InterruptedException e) {
    e.printStackTrace();
    }
    }
    System.out.println("JOB执行结束... ...");
    System.out.println("Hadoop任务ID为:" + submission.getExternalJobId());
    Counters counters =submission.getCounters();
    if(counters != null) {
    System.out.println("计数器:");
    for(CounterGroup group : counters) {
    System.out.print("\t");
    System.out.println(group.getName());
    for(Counter counter : group) {
    System.out.print("\t\t");
    System.out.print(counter.getName());
    System.out.print(": ");
    System.out.println(counter.getValue());
    }
    }
    }
    if(submission.getError().getErrorSummary() != null) {
    System.out.println("JOB执行异常，异常信息为 : " +submission.getError());
    }
    System.out.println("HDFS通过sqoop传输数据到MySQL统计执行完毕");
    }
    }


上边的不适合工作环境
工作环境写一个

String url =”http://ip:12000/sqoop/”;
SqoopClient client = new SqoopClient(url);

client.startJob(jobId)

“`
这个jobid 去shell
show job 定时执行就行了.

轻风细雨
关注
专栏目录
安装impala报错python-setuptools is needed by impala-kudu-shell-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86
天净沙闲云
11-08 1752
安装impala的rpm -ivh impala-kudu-shell-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm报错 [root@node-3 soft]# rpm -ivh impala-kudu-shell-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm warning: impal...
sqoop2-1.99.5-cdh5.6.0.tar.gz
01-07
很好用，sqoop2,希望能帮助到你，Sqoop是一个用来将Hadoop和关系型数据库中的数据相互转移的工具
sqoop2-1.99.5-cdh5.5.4.tar.gz的部署搭建
weixin_30338481的博客
05-07 144
　　不多说，直接上干货！ 　　 　　首先，说下博主我，为什么，好端端的Sqoop1用的好好的，然后又安装和学习Sqoop2？ 　　因为，在Cloudera Hue里的Sqoop，是需要Sqoop2。 HUE配置文件hue.ini 的sqoop模块详解（图文详解）（分HA集群） Sqoop2安装简介 　　sqoop2的安装分为server端...
Ubuntu14环境中安装Sqoop1.99.5
阿狸的博客
01-10 275
环境：Ubuntu14.04 64位 alluxio-1.3.0安装位置：/usr/local/alluxio-1.3.0 hadoop2.7.1安装位置：/usr/local/hadoop Sqoop1.99.5安装到/usr/local/sqoop： 1.下载并解压sqoop源程序 sudo tar -zvxf sqoop-1.99.5-bin-hadoop200.tar.g
sqoop.1.99.5
05-05
sqoop.1.99.5 先如今与sqoop1.4完全不一样。二种不同的启动方式
CDH中安装使用sqoop2
天天臭乐呵
05-09 7397
由于我们的Hadoop、Hive等集群都是通过CDH安装部署的，而且CDH本身支持sqoop安装，因此直接就在这里安装测试了。CDH版本：5.14.x 安装过程如下：点击首页的“Cluster 1”进入Cluster 1界面后，点击Clusters->Add Service在Service列表中找到sqoop：这里我们可以看到有Sqoop 1 Client和Sqoop2，并且Sqoop2的描...
CDH-5.9.0-1.cdh5.9.0.p0.23-el7.parcel.tar.gz.aa
02-26
大数据组件，官方版本：CDH5.9.0 操作系统：CentOS7/RHEL7 完整系列： CDH-5.9.0-1.cdh5.9.0.p0.23-el7.parcel（该文件被分割成三个压缩包，例如：.aa/.ab.ac） CDH-5.9.0-1.cdh5.9.0.p0.23-el7.parcel.sha 由于 ...
cdh版本sqoop1.99.5 api调用
11-23
使用cdh版本的sqoop2 api远程操作sqoop进行数据迁移。
CDH-5.9.0-1.cdh5.9.0.p0.23-el7.parcel.tar.gz.ac
02-26
大数据组件，官方版本：CDH5.9.0 操作系统：CentOS7/RHEL7 完整系列： CDH-5.9.0-1.cdh5.9.0.p0.23-el7.parcel（该文件被分割成三个压缩包，例如：.aa/.ab.ac） CDH-5.9.0-1.cdh5.9.0.p0.23-el7.parcel.sha 由于 ...
CDH-5.9.0-1.cdh5.9.0.p0.23-el7.parcel.tar.gz.ab
02-26
大数据组件，官方版本：CDH5.9.0 操作系统：CentOS7/RHEL7 完整系列： CDH-5.9.0-1.cdh5.9.0.p0.23-el7.parcel（该文件被分割成三个压缩包，例如：.aa/.ab.ac） CDH-5.9.0-1.cdh5.9.0.p0.23-el7.parcel.sha 由于 ...
sqoop-connector-generic-jdbc-1.99.7.jar
08-04
sqoop-connector-generic-jdbc-1.99.7.jar sqoop-connector-generic-jdbc-1.99.7.jar
CDH的sqoop2，hue，oozie集成
wcl722109的博客
09-05 650
压缩所有cdh的jar包 操作： jar cv0f cdh-libs.jar -C /opt/cloudera/parcels/CDH/jars/ . 上传到hdfs sudo -u hue hadoop fs -put cdh-libs.jar /user/hue/ 上传mysql驱动到hdfs sudo -u hue hdfs dfs -put mysql-connecto...
Hadoop2.7.3环境下Sqoop1.99.5安装
qq_38799155的博客
09-01 1280
一、安装准备1.已经装好的hadoop环境是hadoop 2.7.3 参考博客：http://blog.csdn.net/qq_38799155/article/details/777488312.下载sqoop的安装包（注意是hadoop200） 下载地址：http://archive.apache.org/dist/sqoop/1.99.5/ 二、搭建环境1.通过xftp，将下载的sqoo
cdh jar包 sqoop2_CDH添加Sqoop2
weixin_35884854的博客
12-28 202
选择集群，添加服务，选择Sqoop2，选择要安装Sqoop2的服务器，一路继续下去即可。查找 服务端启动命令sqoop2-serverfind / -name sqoop2-servercd /opt/cloudera/parcels/CDH-5.13.2-1.cdh5.13.2.p0.3/bin/./sqoop2-server start查找客户端连接 sqoop2 命令find / -name...
CDH SQOOP 2实例
liuwei063608的专栏
12-13 4270
1. CDH sqoop2环境搭建 添加服务，选择sqoop2，然后下一步，遇到错误修改错误。 2. 脚本命令查看 参考 http://sqoop.apache.org/ http://sqoop.apache.org/docs/1.99.5/Sqoop5MinutesDemo.html   2.1 查看连接 show connector   CDH5.5
CDH集成Sqoop2做全库导入
大壮的博客
05-24 1144
Sqoop简介 Sqoop是一个用于Hadoop和关系型数据库或主机之间的数据传输工具。它可以将数据从关系型数据库import到HDFS，也可以从HDFS export到关系型数据库，通过Hadoop的MapReduce实现。 我们现在的需求就是需要sqoop导入，真个库从mysql到hive里面咯。 Sqoop命令 执行 sqoop help, 可以看到Sqoop支持的命令： Sq...
sqoop 1.99.5安装
csdn问鼎
05-05 344
1.之前安装了1.99.5版本，不是很好用，重新换了个版本 2.下载1.99.5安装包 3.目录下sqoop-1.99.5/server/conf 文件sqoop.properties，修改配置文件org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/opt/soft/hadoop-2.7.5/etc/hado
Sqoop2 使用
afunyusong的专栏
01-12 727
Sqoop2的安装与使用 使用Sqoop从MySQL导入数据到Hive和HBase 及近期感悟 Sqoop User Guide 问题1： java.lang.NoClassDefFoundError: org/codehaus/jackson/map/JsonMappingException 解决办法： cd /opt/cloudera/parcels/CDH/lib/hadoop/
CDH6.3.2集群部署5—添加sqoop
一点见解
05-14 820
添加服务 分配角色 测试 #将pwd替换成master节点安装的数据库root密码 sqoop list-databases --connect jdbc:mysql://dbnode1:3306/ --username root --password pwd
VMware下CDH 5.9.0测试环境三节点Hadoop集群部署教程
该文档详细介绍了在VMware Workstation Pro 12环境中搭建CDH 5.9.0大数据集群的步骤，以供测试环境使用，尤其适合本地开发和维护工作。部署方案采用虚拟化技术，共规划了三台虚拟机：一台为主节点（cdh01），配置为...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值