记一次 sqoop2 shell 转移mysql 数据到hdfs 爬坑之路（步骤详细！！！）

出现问题汇总

• GENERIC_JDBC_CONNECTOR_0003:Unable to access meta data -
• GENERIC_JDBC_CONNECTOR_0016:Can’t fetch schema -
• GENERIC_JDBC_CONNECTOR_0001:Unable to get a connection -
• GENERIC_HDFS_CONNECTOR_0007:Invalid input/output directory - Unexpected exception。。。Caused by: Exception: java.lang.Throwable Message: User: root is not allowed to impersonate root

具体过程流水账

进入sqoop2 shell环境：

[root@master ~]# sqoop2-shell
Setting conf dir: /usr/local/soft/sqoop/bin/../conf
Sqoop home directory: /usr/local/soft/sqoop
Sqoop Shell: Type 'help' or '\h' for help.

sqoop:000> 

开始创建mysql 导入的link:

sqoop:000> create link -c generic-jdbc-connector
0    [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Creating link for connector with name generic-jdbc-connector
Please fill following values to create new link object
Name: mysqllink

Database connection

Driver class: com.mysql.jdbc.Driver
Connection String: jdbc:mysql://localhost:3306/app_central
Username: user01
Password: 
Fetch Size:
Connection Properties:
There are currently 0 values in the map:
entry# protocol-tcp
Error message: Don't know what to do with protocol-tcp
There are currently 0 values in the map:
entry# protocol=tcp
There are currently 1 values in the map:
protocol = tcp
entry#

SQL Dialect

Identifier enclose:
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
New link was successfully created with validation status OK and name mysqllink

创建导出到hdfs 的link:

sqoop:000> create link -c hdfs-connector
Creating link for connector with name hdfs-connector
Please fill following values to create new link object
Name: hdfslink

HDFS cluster

URI: hdfs://master:9000
Conf directory:
Additional configs::
There are currently 0 values in the map:
entry#
New link was successfully created with validation status OK and name hdfslink

查看已创建的link 列表：

sqoop:000> show link
+-----------+------------------------+---------+
|   Name    |     Connector Name     | Enabled |
+-----------+------------------------+---------+
| mysqllink | generic-jdbc-connector | true    |
| hdfslink  | hdfs-connector         | true    |
+-----------+------------------------+---------+

创建数据从mysql 转移到hdfs 的job :

sqoop:000> create job -f mysqllink -t hdfslink
Creating job for links with from name mysqllink and to name hdfslink
Please fill following values to create new job object
Name: mysqljob

Database source

Schema name: app_central
Table name: device
SQL statement:
Column names:
There are currently 0 values in the list:
element#
Partition column:
Partition column nullable:
Boundary query:

Incremental read

Check column:
Last value:

Target configuration

Override null value:
Null value:
File format:
  0 : TEXT_FILE
  1 : SEQUENCE_FILE
  2 : PARQUET_FILE
Choose: 0
Compression codec:
  0 : NONE
  1 : DEFAULT
  2 : DEFLATE
  3 : GZIP
  4 : BZIP2
  5 : LZO
  6 : LZ4
  7 : SNAPPY
  8 : CUSTOM
Choose: 0
Custom codec:
Output directory: /tets
Append mode:

Throttling resources

Extractors:
Loaders:

Classpath configuration

Extra mapper jars:
There are currently 0 values in the list:
element#
New job was successfully created with validation status OK  and name mysqljob

运行job & 排错过程

开始运行job :

sqoop:000> start job -n mysqljob
Exception has occurred during processing command
Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_JDBC_CONNECTOR_0003:Unable to access meta data -

第一个错误：[0003]

GENERIC_JDBC_CONNECTOR_0003:Unable to access meta data -

随后排查时，发现sqoop2 shell 默认是连接在本机的12000 端口的sqoop2 server 上，但是排查发现本机12000 未绑定监听，之后将sqoop.properties 配置文件的端口设置为12000, 重新启动sqoop2 服务，这次正常绑定12000 端口。

随后重启sqoop2 shell ，