Apache Sqoop 安装与配置

 Apache Sqoop 安装与配置

18.1. 安装 Sqoop

OSCM 一键安装

curl -s https://raw.githubusercontent.com/oscm/shell/master/database/apache-sqoop/sqoop-1.99.7-bin-hadoop200.sh | bash
		

启动 Sqoop

/srv/apache-sqoop/bin/sqoop.sh server start
		

检查 Sqoop 线程

		
[hadoop@netkiller ~]$ jps
2512 SecondaryNameNode
23729 SqoopJettyServer
2290 DataNode
871 ResourceManager
23885 Jps

18.2. sqoop2-tool

18.2.1. verify

			
[hadoop@iZj6ciilv2rcpgauqg2uuwZ ~]$ sqoop2-tool verify
Setting conf dir: /srv/apache-sqoop/bin/../conf
Sqoop home directory: /srv/apache-sqoop
Sqoop tool executor:
	Version: 1.99.7
	Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
	Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
6    [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.			
			
			

18.2.2. upgrade

			
[hadoop@iZj6ciilv2rcpgauqg2uuwZ apache-hadoop]$ sqoop2-tool upgrade
Setting conf dir: /srv/apache-sqoop/bin/../conf
Sqoop home directory: /srv/apache-sqoop
Sqoop tool executor:
	Version: 1.99.7
	Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
	Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.UpgradeTool
0    [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
Tool class org.apache.sqoop.tools.tool.UpgradeTool has finished correctly.			
			
			

18.3. sqoop2-shell

进入 sqoop2-shell

		
[hadoop@netkiller ~]$ sqoop2-shell 
Setting conf dir: /srv/apache-sqoop/bin/../conf
Sqoop home directory: /srv/apache-sqoop
Sqoop Shell: Type 'help' or '\h' for help.

sqoop:000>
		
		

Sqoop client script:

		
sqoop2-shell /path/to/your/script.sqoop
		
		

18.3.1. show version

			
sqoop:000> show version
client version:
  Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb 
  Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016
			
			

			
sqoop:000> show version --all 
client version:
  Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb 
  Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016
0    [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
server version:
  Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb 
  Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016
API versions:
  [v1]			
			
			

18.3.2. set

18.3.2.1. server

				
sqoop:000> set server --host master --port 12000 --webapp sqoop
Server is set successfully			
				
				

18.3.2.2. 要设置可查看具体出错信息

				
sqoop:000> set option --name verbose --value true
Verbose option was changed to true				
				
				

18.3.3. show connector

sqoop:000> show connector
0    [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
+------------------------+---------+------------------------------------------------------------+----------------------+
|          Name          | Version |                           Class                            | Supported Directions |
+------------------------+---------+------------------------------------------------------------+----------------------+
| generic-jdbc-connector | 1.99.7  | org.apache.sqoop.connector.jdbc.GenericJdbcConnector       | FROM/TO              |
| kite-connector         | 1.99.7  | org.apache.sqoop.connector.kite.KiteConnector              | FROM/TO              |
| oracle-jdbc-connector  | 1.99.7  | org.apache.sqoop.connector.jdbc.oracle.OracleJdbcConnector | FROM/TO              |
| ftp-connector          | 1.99.7  | org.apache.sqoop.connector.ftp.FtpConnector                | TO                   |
| hdfs-connector         | 1.99.7  | org.apache.sqoop.connector.hdfs.HdfsConnector              | FROM/TO              |
| kafka-connector        | 1.99.7  | org.apache.sqoop.connector.kafka.KafkaConnector            | TO                   |
| sftp-connector         | 1.99.7  | org.apache.sqoop.connector.sftp.SftpConnector              | TO                   |
+------------------------+---------+------------------------------------------------------------+----------------------+
sqoop:000>			
sqoop list-databases --connect  jdbc:mysql://192.168.1.1:3306/ --username root --password 123456
			

			
sqoop:000> show connector --all
7 connector(s) to show: 
Connector with Name: generic-jdbc-connector 
  Class: 1.99.7
  Version: FROM/TO
  Supported Directions {4}
    link config 1:
      Name: linkConfig
      Label: Database connection
      Help: Contains configuration that is required to establish connection with your database server.
      Input 1:
        Name: linkConfig.jdbcDriver
        Label: Driver class
        Help: Fully qualified class name of the JDBC driver that will be used for establishing this connection. Check documentation for instructions how to make the driver's jar files available to Sqoop 2 server.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 128
      Input 2:
        Name: linkConfig.connectionString
        Label: Connection String
        Help: JDBC connection string associated with your database server.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 2000
      Input 3:
        Name: linkConfig.username
        Label: Username
        Help: Username to be used for connection to the database server.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 40
      Input 4:
        Name: linkConfig.password
        Label: Password
        Help: Password to be used for connection to the database server.
        Type: STRING
        Sensitive: true
        Editable By: ANY
        Overrides: 
        Size: 40
      Input 5:
        Name: linkConfig.fetchSize
        Label: Fetch Size
        Help: Optional hint specifying requested JDBC fetch size.
        Type: INTEGER
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 6:
        Name: linkConfig.jdbcProperties
        Label: Connection Properties
        Help: Key-value pairs that should be passed down to JDBC driver when establishing connection.
        Type: MAP
        Sensitive: false
        Editable By: ANY
        Overrides: 
    link config 2:
      Name: dialect
      Label: SQL Dialect
      Help: Database dialect that should be used for generated queries.
      Input 1:
        Name: dialect.identifierEnclose
        Label: Identifier enclose
        Help: Character(s) that should be used to enclose table name, schema or column names.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 5
    FROM Job config 1:
      Name: fromJobConfig
      Label: Database source