问题
- 执行sqoop job要将mysql中数据导入hive表,执行到连接hiveserver2时就卡住不动
- 单独执行连接hiveserver2可以正常连接
Import options:
sqoop import --connect jdbc:mysql://xxx:3306/test --username xxxxx --password xxxxxx
--table t_user
--fields-terminated-by '\001'
--hive-import
--hive-database sqoop2hive
--hive-table t_user_hive
--hive-overwrite
--delete-target-dir --m 1
Output:
......
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/07/27 11:49:12 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7.7.1.1.0-565
20/07/27 11:49:12 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
20/07/27 11:49:13 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
20/07/27 11:49:13 INFO tool.CodeGenTool: Beginning code generation
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
......
20/07/27 11:49:15 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/bf26bf5b6aa581837cbdf0d60c4867e1/t_user.jar
20/07/27 11:49:16 INFO tool.ImportTool: Destination directory t_user is not present, hence not deleting.
20/07/27 11:49:16 WARN manager.MySQLManager: It looks like you are importing from mysql.
20/07/27 11:49:16 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
......
20/07/27 11:49:16 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
20/07/27 11:49:16 INFO client.RMProxy: Connecting to ResourceManager at xxx/10.23.98.75:8032
20/07/27 11:49:16 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/root/.staging/job_1595209123778_0044
20/07/27 11:49:19 INFO db.DBInputFormat: Using read commited transaction isolation
......
20/07/27 11:49:19 INFO impl.YarnClientImpl: Submitted application application_1595209123778_0044
20/07/27 11:49:19 INFO mapreduce.Job: The url to track the job: http://xxx:8088/proxy/application_1595209123778_0044/
20/07/27 11:49:19 INFO mapreduce.Job: Running job: job_1595209123778_0044
20/07/27 11:49:26 INFO mapreduce.Job: Job job_1595209123778_0044 running in uber mode : false
20/07/27 11:49:26 INFO mapreduce.Job: map 0% reduce 0%
20/07/27 11:49:32 INFO mapreduce.Job: map 100% reduce 0%
20/07/27 11:49:32 INFO mapreduce.Job: Job job_1595209123778_0044 completed successfully
20/07/27 11:49:33 INFO mapreduce.Job: Counters: 33
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=248550
FILE: Number of read operations=0
......
HDFS: Number of write operations=2
HDFS: Number of bytes read erasure-coded=0
Job Counters
Launched map tasks=1
Other local map tasks=1
.......
Total megabyte-milliseconds taken by all map tasks=3757056
Map-Reduce Framework
Map input records=9
......
Peak Map Virtual memory (bytes)=2851627008
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=104
20/07/27 11:49:33 INFO mapreduce.ImportJobBase: Transferred 104 bytes in 16.5599 seconds (6.2802 bytes/sec)
20/07/27 11:49:33 INFO mapreduce.ImportJobBase: Retrieved 9 records.
20/07/27 11:49:33 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t_user` AS t LIMIT 1
20/07/27 11:49:33 INFO hive.HiveImport: Loading uploaded data into Hive
20/07/27 11:49:33 INFO hive.HiveImport: SLF4J: Class path contains multiple SLF4J bindings.
20/07/27 11:49:33 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.1-1.cdh7.1.1.p0.3266817/jars/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
20/07/27 11:49:33 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.1-1.cdh7.1.1.p0.3266817/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
......
20/07/27 11:49:35 INFO hive.HiveImport: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
20/07/27 11:49:35 INFO hive.HiveImport: SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
20/07/27 11:49:37 INFO hive.HiveImport: Connecting to jdbc:hive2://xxx:2181,xxx:2181,xxx:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
就停在这:
20/07/27 11:49:37 INFO hive.HiveImport:Connecting to jdbc:hive2://xxx:2181,xxx:2181,xxx:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
连接不上 hiveserver2。
解决
在hive conf目录(一般在/etc/hive/conf)新建(如果没有)一个beeline-hs2-connection.xml文件:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>beeline.hs2.connection.user</name>
<value>hive</value>
</property>
<property>
<name>beeline.hs2.connection.password</name>
<value>hive</value>
</property>
</configuration>