linux查看sqoop目录,Sqoop安装和简单使用

下载sqoop

因为官方并不建议在生产环境中使用sqoop2，即1.99.7，所以这里我用的是sqoop1,即1.4.7

点击下载：http://mirror.bit.edu.cn/apache/sqoop/

解压和上传

我用的xshell辅助工具，先在本地解压改了名称之后上传到Linux上的

sqoop.png

上传linux

linux下.png

也可上传压缩包，执行解压缩，执行命令：

cd /home/hadoop

tar -xvf sqoop-1.4.7.bin__hadoop-2.6.0.tar

配置环境变量

1.配置环境变量

编辑/etc/profile文件，添加SQOOP_HOME变量，并且将$SQOOP_HOME/bin添加到PATH变量中

添加后的profile文件内容

export JAVA_HOME=/home/hadoop/jdk1.8.0_144

export HADOOP_HOME=/home/hadoop/hadoop-2.7.2

export HBASE_HOME=/home/hadoop/hbase-1.2.2

export HIVE_HOME=/home/hadoop/hive-2.1.0

export SQOOP_HOME=/home/hadoop/sqoop-1.4.7

export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$HIVE_HOME/bin:$SQOOP_HOME/bin:$PATH

编辑完成后，执行命令： source /etc/profile

2.sqoop配置文件修改

2.1 sqoop-env.sh 文件

进入 /home/hadoop/sqoop-1.4.7/conf 目录下，也就是执行：

cd /home/hadoop/sqoop-1.4.7/conf

将sqoop-env-template.sh复制一份，并取名为sqoop-env.sh，也就是执行命令：

cp sqoop-env-template.sh sqoop-env.sh

文件末尾加入一下配置：

#Set the path for where zookeper config dir is

#export ZOOCFGDIR=

export HADOOP_COMMON_HOME=/home/hadoop/hadoop-2.7.2

export HADOOP_MAPRED_HOME=/home/hadoop/hadoop-2.7.2

export HIVE_HOME=/home/hadoop/hive-2.1.0

export HBASE_HOME=/home/hadoop/hbase-1.2.2

注：上面的路径要改为自己的路径

3.把MySQL的驱动包上传到sqoop的lib下

驱动包.png

本人测试这个版本在这里是可以的

4.使用sqoop

安装后，如果命令不涉及hive和hdfs的，可以不启动，例如sqoop help命令：

此命令帮助查看sqoop有哪些命令

[hadoop@master ~]$ sqoop help

Warning: /home/hadoop/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.

Please set $HCAT_HOME to the root of your HCatalog installation.

Warning: /home/hadoop/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

Warning: /home/hadoop/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.

Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.

18/02/24 16:39:16 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7

usage: sqoop COMMAND [ARGS]

Available commands:

codegen Generate code to interact with database records

create-hive-table Import a table definition into Hive

eval Evaluate a SQL statement and display the results

export Export an HDFS directory to a database table

help List available commands

import Import a table from a database to HDFS

import-all-tables Import tables from a database to HDFS

import-mainframe Import datasets from a mainframe server to HDFS

job Work with saved jobs

list-databases List available databases on a server

list-tables List available tables in a database

merge Merge results of incremental imports

metastore Run a standalone Sqoop metastore

version Display version information

See 'sqoop help COMMAND' for information on a specific command.

5.使用sqoop查看mysql中的数据表：

进入$SQOOP_HOME/bin目录下执行如下命令：连接mysql看有多少个表

./sqoop list-databases --connect jdbc:mysql://192.168.1.34:3306/test?characterEncoding=UTF-8 --username root --password '123'

命令.png

6.把MySQL中的表导入hdfs中

前提：一定要启动hdfs和yarn,本人忘了启动yarn，一直显示连接超时，搞了半天才反应过来.......

sqoop import -m 1 --connect jdbc:mysql://192.168.1.34:3306/test --username root --password 123 --table name --target-dir /user/sqoop/datatest

显示：

18/02/24 15:52:32 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar

18/02/24 15:52:33 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps

18/02/24 15:52:33 INFO client.RMProxy: Connecting to ResourceManager at master.hadoop/192.168.139.128:8032

18/02/24 15:52:38 INFO db.DBInputFormat: Using read commited transaction isolation

18/02/24 15:52:38 INFO mapreduce.JobSubmitter: number of splits:1

18/02/24 15:52:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1519458728102_0001

18/02/24 15:52:39 INFO impl.YarnClientImpl: Submitted application application_1519458728102_0001

18/02/24 15:52:39 INFO mapreduce.Job: The url to track the job: http://master.hadoop:8088/proxy/application_1519458728102_0001/

18/02/24 15:52:39 INFO mapreduce.Job: Running job: job_1519458728102_0001

18/02/24 15:52:51 INFO mapreduce.Job: Job job_1519458728102_0001 running in uber mode : false

18/02/24 15:52:51 INFO mapreduce.Job: map 0% reduce 0%

18/02/24 15:52:57 INFO mapreduce.Job: map 100% reduce 0%

18/02/24 15:52:57 INFO mapreduce.Job: Job job_1519458728102_0001 completed successfully

18/02/24 15:52:58 INFO mapreduce.Job: Counters: 30

File System Counters

FILE: Number of bytes read=0

FILE: Number of bytes written=136482

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=87

HDFS: Number of bytes written=16

HDFS: Number of read operations=4

HDFS: Number of large read operations=0

HDFS: Number of write operations=2

Job Counters

Launched map tasks=1

Other local map tasks=1

Total time spent by all maps in occupied slots (ms)=3648

Total time spent by all reduces in occupied slots (ms)=0

Total time spent by all map tasks (ms)=3648

Total vcore-milliseconds taken by all map tasks=3648

Total megabyte-milliseconds taken by all map tasks=3735552

Map-Reduce Framework

Map input records=1

Map output records=1

Input split bytes=87

Spilled Records=0

Failed Shuffles=0

Merged Map outputs=0

GC time elapsed (ms)=88

CPU time spent (ms)=630

Physical memory (bytes) snapshot=100278272

Virtual memory (bytes) snapshot=2060853248

Total committed heap usage (bytes)=16719872

File Input Format Counters

Bytes Read=0

File Output Format Counters

Bytes Written=16

18/02/24 15:52:58 INFO mapreduce.ImportJobBase: Transferred 16 bytes in 24.5469 seconds (0.6518 bytes/sec)

18/02/24 15:52:58 INFO mapreduce.ImportJobBase: Retrieved 1 records.

未完待续.....