搭建Sqoop环境

已于 2022-06-24 09:43:21 修改
阅读量512 收藏
点赞数
文章标签： hadoop sqoop hive
于 2022-06-09 15:44:12 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/sun_xo/article/details/125203600
版权

1) Build and deploy
## download sqoop-1.4.7.tar.gz from http://archive.apache.org/dist/
## unpack to ~/work/sqoop-src-1.4.7
## modify project files
$ cd ~/work/sqoop-src-1.4.7
$ diff -u build.xml.orig build.xml  # change to valid repository, omit docs

-- build.xml.orig	2017-12-19 07:00:00.000000000 +0800
+++ build.xml	2022-06-08 17:27:49.000000000 +0800
@@ -171,9 +171,9 @@
   <property name="ivysettings.xml" location="${ivy.dir}/ivysettings.xml"/>
   <property name="ivy.jar" location="${lib.dir}/ivy-${ivy.version}.jar"/>
   <property name="ivy_repo_url"
-  value="http://repo2.maven.org/maven2/org/apache/ivy/ivy/${ivy.version}/ivy-${ivy.version}.jar" />
+  value="https://maven.aliyun.com/repository/public/org/apache/ivy/ivy/${ivy.version}/ivy-${ivy.version}.jar" />
   <property name="mvn_repo_url"
-  value="http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/${mvn.version}/maven-ant-tasks-${mvn.version}.jar"/>
+  value="https://maven.aliyun.com/repository/public/org/apache/maven/maven-ant-tasks/${mvn.version}/maven-ant-tasks-${mvn.version}.jar"/>
   <property name="mvn.jar"
       location="${build.dir}/maven-ant-tasks-${mvn.version}.jar" />
   <property name="build.ivy.dir" location="${build.dir}/ivy" />
@@ -184,7 +184,7 @@
 
   <!--this is the naming policy for artifacts we want pulled down-->
   <property name="ivy.artifact.retrieve.pattern"
-      value="${name}/[conf]/[artifact]-[revision](-[classifier]).[ext]"/>
+      value="${name}/[conf]/[artifact]-[type]-[revision](-[classifier]).[ext]"/>
 
   <!--test related properties -->
   <property name="sqoop.test.oracle.connectstring" value="jdbc:oracle:thin:@//localhost/xe"/>
@@ -491,7 +491,7 @@
   </target>
 
   <target name="package"
-      depends="jar-all,compile-all,docs,ivy-retrieve-redist,scripts"
+      depends="jar-all,compile-all,ivy-retrieve-redist,scripts"
       description="Create a redistributable package">
 
     <mkdir dir="${dist.dir}"/>
@@ -542,7 +542,7 @@
     </copy>
 
     <!-- copy in documentation build artifacts -->
-    <copy todir="${dist.dir}/docs" includeEmptyDirs="false" flatten="false">
+    <!-- copy todir="${dist.dir}/docs" includeEmptyDirs="false" flatten="false">
       <fileset dir="${build.dir}/docs">
         <include name="**/*.html" />
         <include name="**/*.css" />
@@ -553,7 +553,7 @@
       <fileset dir="${build.dir}/docs">
         <include name="**/*.gz" />
       </fileset>
-    </copy>
+    </copy -->
 
     <!-- copy in auto-generated bin scripts -->
     <copy todir="${dist.dir}/bin" includeEmptyDirs="false" flatten="true">

$ diff -u ivy/ivysettings.xml.orig ivy/ivysettings.xml # change repository

--- ivy/ivysettings.xml.orig	2017-12-19 07:00:00.000000000 +0800
+++ ivy/ivysettings.xml	2022-06-08 17:29:13.000000000 +0800
@@ -31,7 +31,7 @@
           http://ibiblio.lsu.edu/main/pub/packages/maven2
           http://www.ibiblio.net/pub/packages/maven2
   -->
-  <property name="repo.maven.org" value="http://repo1.maven.org/maven2/"
+  <property name="repo.maven.org" value="https://maven.aliyun.com/repository/public/"
       override="false"/>
   <property name="snapshot.apache.org"
       value="https://repository.apache.org/content/repositories/snapshots/"

$ diff ivy/libraries.properties.orig ivy/libraries.properties # change hadoop version

--- ivy/libraries.properties.orig	2017-12-19 07:00:00.000000000 +0800
+++ ivy/libraries.properties	2022-06-08 17:29:00.000000000 +0800
@@ -54,6 +54,6 @@
 
 slf4j.version=1.7.7
 
-hadoop.version=2.6.0
+hadoop.version=2.8.0
 hbase.version=1.2.4
 hcatalog.version=1.2.1

## build, create the tar ball if build sucessful
$ ant tar       
$ cd ~/work
$ tar xvf sqoop-src-1.4.7/build/sqoop-1.4.7.bin__hadoop-2.8.0.tar.gz
$ mv sqoop-1.4.7.bin__hadoop-2.8.0 sqoop-1.4.7

## copy mysql driver
$ cp -p dwonloads/mysql-connector-java-8.0.28.jar sqoop-1.4.7/lib/

## customize environment
$ cd ~/work/sqoop-1.4.7
$ diff -u conf/sqoop-env-template.sh conf/sqoop-env.sh

--- conf/sqoop-env-template.sh	2022-06-08 17:31:21.000000000 +0800
+++ conf/sqoop-env.sh	2022-06-09 09:11:44.000000000 +0800
@@ -20,16 +20,16 @@
 # Set Hadoop-specific environment variables here.
 
 #Set path to where bin/hadoop is available
-#export HADOOP_COMMON_HOME=
+export HADOOP_COMMON_HOME=$HADOOP_HOME
 
 #Set path to where hadoop-*-core.jar is available
-#export HADOOP_MAPRED_HOME=
+export HADOOP_MAPRED_HOME=$HADOOP_HOME
 
 #set the path to where bin/hbase is available
 #export HBASE_HOME=
 
 #Set the path to where bin/hive is available
-#export HIVE_HOME=
+export HIVE_HOME=/Users/sun_xo/work/hive-2.3.9
 
 #Set the path for where zookeper config dir is
 #export ZOOCFGDIR=

2) Import from MySQL
$ cd ~/work/sqoop-1.4.7
## list tables
$ bin/sqoop list-tables --connect jdbc:mysql://localhost:3306/manga --username manga --password manga

...
export
form
form_detail
fruit

$ bin/sqoop import --connect jdbc:mysql://localhost:3306/manga --username manga --password manga \
    --table fruit --target-dir /user/sun_xo/sqoop/fruit \
    --num-mappers 1 --delete-target-dir

22/06/09 15:33:13 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
22/06/09 15:33:13 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
22/06/09 15:33:13 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
22/06/09 15:33:13 INFO tool.CodeGenTool: Beginning code generation
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
22/06/09 15:33:13 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `fruit` AS t LIMIT 1
22/06/09 15:33:14 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `fruit` AS t LIMIT 1
22/06/09 15:33:14 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /Users/sun_xo/work/hadoop
Note: /tmp/sqoop-sun_xo/compile/47ca53cd7c2f9abf409e90ef9e7734d3/fruit.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
22/06/09 15:33:15 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-sun_xo/compile/47ca53cd7c2f9abf409e90ef9e7734d3/fruit.jar
22/06/09 15:33:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/06/09 15:33:16 INFO tool.ImportTool: Destination directory /user/sun_xo/sqoop/fruit deleted.
22/06/09 15:33:16 WARN manager.MySQLManager: It looks like you are importing from mysql.
22/06/09 15:33:16 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
22/06/09 15:33:16 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
22/06/09 15:33:16 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
22/06/09 15:33:16 INFO mapreduce.ImportJobBase: Beginning import of fruit
22/06/09 15:33:16 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
22/06/09 15:33:16 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
22/06/09 15:33:16 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
22/06/09 15:33:16 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
22/06/09 15:33:17 INFO db.DBInputFormat: Using read commited transaction isolation
22/06/09 15:33:17 INFO mapreduce.JobSubmitter: number of splits:1
22/06/09 15:33:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1654591879954_0009
22/06/09 15:33:17 INFO impl.YarnClientImpl: Submitted application application_1654591879954_0009
22/06/09 15:33:17 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1654591879954_0009/
22/06/09 15:33:17 INFO mapreduce.Job: Running job: job_1654591879954_0009
22/06/09 15:33:23 INFO mapreduce.Job: Job job_1654591879954_0009 running in uber mode : false
22/06/09 15:33:23 INFO mapreduce.Job:  map 0% reduce 0%
22/06/09 15:33:29 INFO mapreduce.Job:  map 100% reduce 0%
22/06/09 15:33:29 INFO mapreduce.Job: Job job_1654591879954_0009 completed successfully
22/06/09 15:33:29 INFO mapreduce.Job: Counters: 30
	File System Counters
		FILE: Number of bytes read=0
		FILE: Number of bytes written=156473
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=87
		HDFS: Number of bytes written=68
		HDFS: Number of read operations=4
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=2
	Job Counters 
		Launched map tasks=1
		Other local map tasks=1
		Total time spent by all maps in occupied slots (ms)=2213
		Total time spent by all reduces in occupied slots (ms)=0
		Total time spent by all map tasks (ms)=2213
		Total vcore-milliseconds taken by all map tasks=2213
		Total megabyte-milliseconds taken by all map tasks=2266112
	Map-Reduce Framework
		Map input records=4
		Map output records=4
		Input split bytes=87
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=40
		CPU time spent (ms)=0
		Physical memory (bytes) snapshot=0
		Virtual memory (bytes) snapshot=0
		Total committed heap usage (bytes)=107479040
	File Input Format Counters 
		Bytes Read=0
	File Output Format Counters 
		Bytes Written=68
22/06/09 15:33:29 INFO mapreduce.ImportJobBase: Transferred 68 bytes in 12.5582 seconds (5.4148 bytes/sec)
22/06/09 15:33:29 INFO mapreduce.ImportJobBase: Retrieved 4 records.

$ hdfs dfs -text "/user/sun_xo/sqoop/fruit/*"

101,香瓜,800.0
102,草莓,150.0
103,苹果,120.0
104,柠檬,200.0

3) Export to MySQL
$ cat fruit_exp.txt

105,橙子,115.0
106,香蕉,110.0

$ hdfs dfs -mkdir -p /user/sun_xo/sqoop/export/fruit
$ hdfs dfs -put fruit_exp.txt /user/sun_xo/sqoop/export/fruit
$ bin/sqoop export --connect jdbc:mysql://localhost:3306/manga --username manga --password manga \
    --table fruit --export-dir /user/sun_xo/sqoop/export/fruit \
    --num-mappers 1

22/06/09 15:39:59 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
22/06/09 15:39:59 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
22/06/09 15:39:59 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
22/06/09 15:39:59 INFO tool.CodeGenTool: Beginning code generation
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
22/06/09 15:39:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `fruit` AS t LIMIT 1
22/06/09 15:39:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `fruit` AS t LIMIT 1
22/06/09 15:39:59 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /Users/sun_xo/work/hadoop
Note: /tmp/sqoop-sun_xo/compile/89d2056e28755b6b08867f478c64d7b4/fruit.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
22/06/09 15:40:01 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-sun_xo/compile/89d2056e28755b6b08867f478c64d7b4/fruit.jar
22/06/09 15:40:01 INFO mapreduce.ExportJobBase: Beginning export of fruit
22/06/09 15:40:01 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
22/06/09 15:40:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/06/09 15:40:01 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
22/06/09 15:40:02 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
22/06/09 15:40:02 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
22/06/09 15:40:02 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
22/06/09 15:40:02 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
22/06/09 15:40:02 INFO input.FileInputFormat: Total input files to process : 1
22/06/09 15:40:02 INFO input.FileInputFormat: Total input files to process : 1
22/06/09 15:40:02 INFO mapreduce.JobSubmitter: number of splits:1
22/06/09 15:40:02 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
22/06/09 15:40:03 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1654591879954_0010
22/06/09 15:40:03 INFO impl.YarnClientImpl: Submitted application application_1654591879954_0010
22/06/09 15:40:03 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1654591879954_0010/
22/06/09 15:40:03 INFO mapreduce.Job: Running job: job_1654591879954_0010
22/06/09 15:40:10 INFO mapreduce.Job: Job job_1654591879954_0010 running in uber mode : false
22/06/09 15:40:10 INFO mapreduce.Job:  map 0% reduce 0%
22/06/09 15:40:15 INFO mapreduce.Job:  map 100% reduce 0%
22/06/09 15:40:16 INFO mapreduce.Job: Job job_1654591879954_0010 completed successfully
22/06/09 15:40:16 INFO mapreduce.Job: Counters: 30
	File System Counters
		FILE: Number of bytes read=0
		FILE: Number of bytes written=156162
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=184
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=4
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=0
	Job Counters 
		Launched map tasks=1
		Rack-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=2376
		Total time spent by all reduces in occupied slots (ms)=0
		Total time spent by all map tasks (ms)=2376
		Total vcore-milliseconds taken by all map tasks=2376
		Total megabyte-milliseconds taken by all map tasks=2433024
	Map-Reduce Framework
		Map input records=2
		Map output records=2
		Input split bytes=147
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=37
		CPU time spent (ms)=0
		Physical memory (bytes) snapshot=0
		Virtual memory (bytes) snapshot=0
		Total committed heap usage (bytes)=121110528
	File Input Format Counters 
		Bytes Read=0
	File Output Format Counters 
		Bytes Written=0
22/06/09 15:40:16 INFO mapreduce.ExportJobBase: Transferred 184 bytes in 14.2446 seconds (12.9172 bytes/sec)
22/06/09 15:40:16 INFO mapreduce.ExportJobBase: Exported 2 records.

$ mysql -umanga -pmanga manga
mysql> select * from fruit;

+----------+--------+-------+
| fruit_id | name   | price |
+----------+--------+-------+
|      101 | 香瓜   |   800 |
|      102 | 草莓   |   150 |
|      103 | 苹果   |   120 |
|      104 | 柠檬   |   200 |
|      105 | 橙子   |   115 |
|      106 | 香蕉   |   110 |
+----------+--------+-------+
6 rows in set (0.00 sec)

sun_xo
关注
Sqoop环境搭建
小财迷嘻嘻的博客
09-26 363
目录1、解压安装2、配置环境变量3、修改配置文件4、拷贝JDBC驱动5、去除warning警告6、验证7、测试sqoop连接数据库 前言：安装Sqoop的前提是已经具备Java和Hadoop、Hive、Zookeeper、HBase的环境。 1、解压安装 下载地址：http://archive.cloudera.com/cdh5/cdh/5/ 解压到opt目录下 tar -zxvf sqoop-1.4.6-cdh5.14.2.tar.gz -C /opt 改名： mv sqoop-1.4.6-cdh5.14
菜鸟先飞之sqoop环境搭建以及jar包依赖
望、本归心的博客
09-03 1649
文件准备下载到本地 sqoop-1.4.6-cdh5.14.2.tar.gz 拖入安装目录并解压 tar -zxfsqoop-1.4.6-cdh5.14.2.tar 解压文件重命名 mvsqoop-1.4.6 sqoop 配置环境变量 vi /etc/profile export SQOOP_HOME=/opt/install/sqoop export PATH=$SQOOP_HOME/bin:$PATH 切换至配置文件目录 cd /opt/install/sqoop/conf...
【Sqoop】（一）Sqoop 概述及安装环境搭建
云祁QI
12-23 1368
文章目录一、Sqoop简介二、Sqoop原理三、Sqoop安装3.1 下载并解压3.2 修改配置文件3.3 拷贝JDBC驱动3.4 配置环境变量3.5 验证是否安装成功 一、Sqoop简介 Sqoop是一款开源的工具，主要用于在Hadoop(Hive)与传统的数据库(mysql、postgresql…)间进行数据的传递，可以将一个关系型数据库（例如： MySQL ,Oracle ,Postgre...
Sqoop 的环境搭建
sunbin11220904的博客
07-23 186
一、Sqoop 安装 安装 Sqoop 的前提是已经具备 Java 和 Hadoop、Hive、ZooKeeper、HBase的环境 1.1 下载并解压 1.上传安装包 sqoop-1.4.6-cdh5.14.2.tar.gz 到虚拟机中 2.解压 sqoop 安装包到指定目录 tar -zxf sqoop-1.4.6-cdh5.14.2.tar.gz -C /opt/ 3改名 mv sqoop-1.4.6-cdh5.14.2 sqoop 4.配置环境变量 vi /etc/profile 添加如下内容：
大数据环境搭建———>Sqoop安装与配置
08-03
安装Sqoop需要预先搭建Java环境和Hadoop环境，其安装过程涉及下载、解压、配置驱动和环境变量设置等步骤。一旦安装配置完成，就可以利用Sqoop来执行大规模数据的转移工作，从而满足大数据应用中对于数据处理和分析的...
Sqoop集群搭建.
09-21
Sqoop 集群搭建是指在 Hadoop 集群环境中安装和配置 Sqoop，以实现数据的高效转换。 一、Sqoop 安装 Sqoop 的安装可以分为三步：下载 Sqoop 压缩包、解压缩包、将 Sqoop 拷贝到指定目录下。 首先，下载 Sqoop ...
Sqoop环境搭建和具体应用
共享，共进步
07-22 211
Sqoop的环境搭建 1，准备好 sqoop-1.4.6-cdh5.14.2.tar.gz java-json.jar mysql-connector-java-5.1.27-bin.jar 2，解压安装，配置环境变量 [root@hadoop151 opt]# vi /etc/profile 添加以下内容 export SQOOP_HOME=/opt/sqoop export PATH=$SQOOP_HOME/bin:$PATH 使生效 [root@hadoop151 opt]# source /
搭建Sqoop开发环境
做技术的那些年
12-12 645
1.什么是sqoopSqoop是一款开源的工具，主要用于在Hadoop与传统的数据库(Mysql、oracle)间进行数据的传递，可以将一个关系型数据库（MySQL ,Oracle 等）中的数据导进到Hadoop的HDFS中，也可以将HDFS的数据导进到关系型数据库中,对于某些NoSQL数据库它也提供了连接器。 简单来说Sqoop就是是一个转换工具，用于在关系型数据库与HDFS之间进行数据转换。 S
sqoop1环境搭建
翻滚啊牛宝宝
12-01 364
Sqoop是Apache软件基金会的一款定级开源数据传输工具，用于在Hadoop与关系型数据库之间进行数据传递，他可以将关系型数据中的数据导入到Hadoop的分布式文件系统（HDFS）中，也可以将分布式文件系统中的数据导出到关系型数据库中。 Sqoop的自动化流程，依赖于被导入的数据库表结构。Sqoop使用MapReduce计算框架来完成数据的导入和导出，并提供了并行操作和容错性。 以上引自《...
Sqoop教程（一）Sqoop环境搭建
Java朱老师博客
05-09 1248
一、上传Sqoop到服务器/apps文件集 二、解压： tar -zxvf /apps/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /usr/local 三、建立软连接： ln -s sqoop-1.4.7.bin__hadoop-2.6.0/ sqoop 如图所示： 四、修改环境变量 cd /etc/profile.d/ vi sqoop.sh export SQOOP_HOME=/usr/local/sqoop export PATH=$PATH:$SQOO
Sqoop环境安装
努力的凹凸曼的博客
06-21 846
环境下载 　　首先将下载的 sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz放到 /usr/hadoop/目录下（该目录可以自定义，一般为Hadoop集群安装目录），然后对安装包解压、修改文件名和修改用户权限。 [hadoop@centpy /]$ cd /usr/hadoop/ [hadoop@centpy hadoop]$ ls flume hadoo...
sqoop使用与原理
jeffry_ding的博客
05-09 410
简介 sqoop:sql-to-hadoop。 Sqoop是一个用来将Hadoop和关系型数据库中的数据相互转移的工具，可以将一个关系型数据库中的数据导进到Hadoop的HDFS中，也可以将HDFS的数据导进到关系型数据库中。 sqoop架构图 Sqoop原理（以import为例） Sqoop在import时，需要制定split-by参数。Sqoop根据不同的split-by参数值...
大数据环境搭建3：Apache Sqoop环境搭建
wangping623的专栏
08-08 416
1. 下载sqoop安装包 下载地址http://mirror.bit.edu.cn/apache/sqoop/1.4.7/ 2. 上传sqoop安装包，并安装 解压安装 tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C ../app 3. 配置环境变量 (1) vi /etc/profile export SQOOP_HOME=/h...
sqoop的安装与配置
u011250186的博客
07-19 482
最近需要将MySQL的数据导出到HDFS，所以搜到了sqoop2。跟sqoop1相比，sqoop2的好处是直接使用程序连接到集群上的sqoop，远程操作。流程是需要先创建link也可以理解成要操作的对象，比如一个link是hdfs，一个link是mysql，有了link后需要创建job，创建job需要指定这两个link进行交互，设置from和to的关系，然后执行job就可以了。 安装： 安装真是个大问题，问题简直层出不穷，花了我整整一个晚上才把它勉强弄好，下面记录一下安装路上遇到的坑s。 首先，我..
idea提交job到yarn上运行,遇到一二三BUG记录
xinji
09-04 843
1.Connecting to ResourceManager at /0.0.0.0:8032 19/09/04 10:54:16 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 19/09/04 10:54:19 INFO ipc.Client: Retrying connect to server: 0....
sqoop 报错：Could not load db driver class: com.microsoft.sqlserver.jdbc.SQLServerDriver
小蚯蚓的博客
05-06 4082
一、问题描述 将sql server的数据导入hive，结果报错： Warning: /opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to th...
Sqoop安装教程
weixin_38023225的博客
10-18 184
1 Sqoop简介 Apache Sqoop(TM)是一种旨在有效地在Apache Hadoop和诸如关系数据库等结构化数据存储之间传输大量数据的工具。 Sqoop于2012年3月孵化出来，现在是一个顶级的Apache项目。 2 Sqoop原理 将导入或导出命令翻译成mapreduce程序来实现。 在翻译出的mapreduce中主要是对inputformat和outputformat进行...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值