Sqoop导入数据至hive卡死在hive.HiveImport: Connecting to jdbc:hive2不执行...

最新推荐文章于 2024-10-14 11:27:37 发布
最新推荐文章于 2024-10-14 11:27:37 发布
阅读量264 收藏
点赞数 2
CC 4.0 BY-SA版权
分类专栏： 大数据部署报错及解决 文章标签： 大数据
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/ye_chen_/article/details/142859480

vi etc/hive/3.1.4.0-315/0/beeline-site.xml添加配置user=hdfs;password=hdfs;

注：每次重启hive后这个临时加载出来的配置都需要改，原因是源头配置文件改了不生效。。暂时未找到永久的解决办法

解决：Sqoop导入数据到Hive连接hiveserver2卡住
我的博客
07-27 3236
问题 执行sqoop job要将mysql中数据导入hive表，执行到连接hiveserver2时就卡住不动 单独执行连接hiveserver2可以正常连接 Import options: sqoop import --connect jdbc:mysql://xxx:3306/test --username xxxxx --password xxxxxx --table t_user --fields-terminated-by '\001' --hive-import --hive-data
2019-4-26-已经解决-beeline连接hive2 出现 java.net.ConnectException: Connection refused (Connection refused)
jasonLux
04-26 2799
_ [centos@s101 ~]$ beeline -u jdbc:hive2://s101:10000 which: no hbase in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/soft/jdk/bin:/soft/hadoop/bin:/soft/hadoop/sbin:/soft/zk/bin:/soft/hive/bin...
【Sqoop】Sqoop的使用(连接MySQL、Hive)
HR的博客
03-18 1万+
Sqoop的使用 使用sqoop前需要启动Hadoop，检查Hadoop的运行状态，我们这里使用的是伪分布式 [root@hr conf]# jps 1538 DataNode 2114 NodeManager 1689 SecondaryNameNode 1834 ResourceManager 1421 NameNode 2831 Jps Sqoop连接MySQL需要MySQL的jar包，吧jar包放置Sqoop的lib目录下 先测试Sqoop能否连接MySQL查看存在的数据库 sqoop list-
sqoop 迁移 oracle 数据到 hive ，卡住 Connecting to jdbc:hive2://
Lbg_007的博客
07-02 1289
现象： sqoop 迁移 oracle 数据到 hive ，卡在Connecting to jdbc:hive2://.... 解决： 在 /etc/hive/conf 增加 beeline-hs2-connection.xml 内容如下： <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <propert..
oracle导入初始数据死机,Sqoop导入Oracle数据至hive卡死在hive.HiveImport: Connecting to jdbc:hive2不执行...
weixin_39955154的博客
04-03 714
环境信息：HDP-3.1.4已经下载好odjbc8.jar驱动程序放置在/usr/hdp/current/sqoop-client/lib/目录Sqoop读取Oracle数据库数据导入Hive时，一直卡在INFO hive.HiveImport: Connecting to jdbc:hive2:// 不懂，日志执行到下面这样子就不执行了。20/10/19 05:16:56 WARN hive.T...
【大数据开发运维解决方案】Sqoop全量同步mysql/Oracle数据到hive
热门推荐
赵延东的一亩三分地
03-19 17万+
接下来本篇文章详细介绍一下使用sqoop全量同步oracle/mysql数据到hive，这里实验采用oracle数据库为例，后面一篇文章将详细介绍：1、sqoop --incremental append 附加模式增量同步数据到hive2、sqoop --incremental --merge-key合并模式增量同步到hive。
sqoop导入关系表到HIVE报错
bitmao888的博客
04-24 1064
第一步：拷贝jar包 将我们mysql表当中的数据直接导入到hive表中的话，我们需要将hive的一个叫做hive-exec-1.1.0-cdh5.14.0.jar的jar包拷贝到sqoop的lib目录下 cp /export/servers/hive-1.1.0-cdh5.14.0/lib/hive-exec-1.1.0-cdh5.14.0.jar /export/servers/sqoop-1...
SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/installs/hadoop3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/installs/hive3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/installs/hbase2.2.2/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2025-10-15 20:23:12,448 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 2025-10-15 20:23:12,530 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 2025-10-15 20:23:12,531 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override 2025-10-15 20:23:12,531 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc. 2025-10-15 20:23:12,778 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 2025-10-15 20:23:12,794 INFO tool.CodeGenTool: Beginning code generation Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 2025-10-15 20:23:14,590 INFO manager.SqlManager: Executing SQL statement: select region_code,region_code_desc,region_city,region_city_desc,region_province,region_province_desc from area_dim where (1 = 0) 2025-10-15 20:23:14,636 INFO manager.SqlManager: Executing SQL statement: select region_code,region_code_desc,region_city,region_city_desc,region_province,region_province_desc from area_dim where (1 = 0) 2025-10-15 20:23:14,699 INFO manager.SqlManager: Executing SQL statement: select region_code,region_code_desc,region_city,region_city_desc,region_province,region_province_desc from area_dim where (1 = 0) 2025-10-15 20:23:14,717 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/installs/hadoop3.3.1 注: /tmp/sqoop-root/compile/2779abb8eed61751692c37913db4d94d/QueryResult.java使用或覆盖了已过时的 API。 注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。 2025-10-15 20:23:18,787 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/2779abb8eed61751692c37913db4d94d/QueryResult.jar 2025-10-15 20:23:20,876 INFO tool.ImportTool: Destination directory /data/nshop/ods/dim_pub_area is not present, hence not deleting. 2025-10-15 20:23:20,881 INFO mapreduce.ImportJobBase: Beginning query import. 2025-10-15 20:23:20,883 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2025-10-15 20:23:20,894 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 2025-10-15 20:23:20,928 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 2025-10-15 20:23:21,530 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at caiji/192.168.193.54:8032 2025-10-15 20:23:21,916 INFO client.AHSProxy: Connecting to Application History server at caiji/192.168.193.54:10200 2025-10-15 20:23:22,262 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1760505557106_0013 Hive Session ID = e086f596-6c65-405f-87f6-0d09bca90e4a Hive Session ID = fa8aa012-e2d9-4c4b-8fbf-ea3bbc7b7e89 2025-10-15 20:23:27,485 INFO db.DBInputFormat: Using read commited transaction isolation 2025-10-15 20:23:27,565 INFO mapreduce.JobSubmitter: number of splits:1 2025-10-15 20:23:28,128 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1760505557106_0013 2025-10-15 20:23:28,129 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2025-10-15 20:23:28,587 INFO conf.Configuration: resource-types.xml not found 2025-10-15 20:23:28,591 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2025-10-15 20:23:28,742 INFO impl.YarnClientImpl: Submitted application application_1760505557106_0013 2025-10-15 20:23:28,819 INFO mapreduce.Job: The url to track the job: http://caiji:8088/proxy/application_1760505557106_0013/ 2025-10-15 20:23:28,820 INFO mapreduce.Job: Running job: job_1760505557106_0013 2025-10-15 20:23:43,166 INFO mapreduce.Job: Job job_1760505557106_0013 running in uber mode : false 2025-10-15 20:23:43,168 INFO mapreduce.Job: map 0% reduce 0% 2025-10-15 20:23:53,451 INFO mapreduce.Job: map 100% reduce 0% 2025-10-15 20:23:53,474 INFO mapreduce.Job: Job job_1760505557106_0013 completed successfully 2025-10-15 20:23:53,720 INFO mapreduce.Job: Counters: 33 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=282339 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=87 HDFS: Number of bytes written=33222 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 HDFS: Number of bytes read erasure-coded=0 Job Counters Launched map tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=7085 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=7085 Total vcore-milliseconds taken by all map tasks=7085 Total megabyte-milliseconds taken by all map tasks=7255040 Map-Reduce Framework Map input records=791 Map output records=791 Input split bytes=87 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=102 CPU time spent (ms)=4340 Physical memory (bytes) snapshot=249790464 Virtual memory (bytes) snapshot=2819796992 Total committed heap usage (bytes)=238026752 Peak Map Physical memory (bytes)=249790464 Peak Map Virtual memory (bytes)=2819796992 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=33222 2025-10-15 20:23:53,733 INFO mapreduce.ImportJobBase: Transferred 32.4434 KB in 32.7752 seconds (1,013.6317 bytes/sec) 2025-10-15 20:23:53,742 INFO mapreduce.ImportJobBase: Retrieved 791 records. 2025-10-15 20:23:53,742 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table null 2025-10-15 20:23:53,795 INFO manager.SqlManager: Executing SQL statement: select region_code,region_code_desc,region_city,region_city_desc,region_province,region_province_desc from area_dim where (1 = 0) 2025-10-15 20:23:53,804 INFO manager.SqlManager: Executing SQL statement: select region_code,region_code_desc,region_city,region_city_desc,region_province,region_province_desc from area_dim where (1 = 0) 2025-10-15 20:23:53,826 INFO hive.HiveImport: Loading uploaded data into Hive 2025-10-15 20:23:53,888 INFO conf.HiveConf: Found configuration file file:/opt/installs/hive3.1.2/conf/hive-site.xml 2025-10-15 20:24:00,619 main ERROR Could not register mbeans java.security.AccessControlException: access denied ("javax.management.MBeanTrustPermission" "register") at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) at java.lang.SecurityManager.checkPermission(SecurityManager.java:585) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanTrustPermission(DefaultMBeanServerInterceptor.java:1848) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:322) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) at org.apache.logging.log4j.core.jmx.Server.register(Server.java:389) at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:167) at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:140) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:556) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:261) at org.apache.logging.log4j.core.async.AsyncLoggerContext.start(AsyncLoggerContext.java:87) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:240) at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:158) at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:131) at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:101) at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:188) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jDefault(LogUtils.java:173) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:106) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:98) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:81) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:331) at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) Hive Session ID = a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:00,804 INFO SessionState: Hive Session ID = a3156ff3-c06b-4333-9294-e6fd634a4b9d Logging initialized using configuration in jar:file:/opt/installs/sqoop1.4.7/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true 2025-10-15 20:24:01,091 INFO SessionState: Logging initialized using configuration in jar:file:/opt/installs/sqoop1.4.7/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true 2025-10-15 20:24:01,206 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:01,512 INFO session.SessionState: Created local directory: /opt/installs/hive3.1.2/iotmp/root/a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:01,523 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/a3156ff3-c06b-4333-9294-e6fd634a4b9d/_tmp_space.db 2025-10-15 20:24:01,557 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:01,557 INFO session.SessionState: Updating thread name to a3156ff3-c06b-4333-9294-e6fd634a4b9d main 2025-10-15 20:24:06,029 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://caiji:9083 2025-10-15 20:24:06,217 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 1 2025-10-15 20:24:06,321 INFO metastore.HiveMetaStoreClient: Connected to metastore. 2025-10-15 20:24:06,322 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=root (auth:SIMPLE) retries=1 delay=1 lifetime=0 Hive Session ID = 32336280-d41f-4a0a-b3d7-b39399f2bbab 2025-10-15 20:24:07,603 INFO SessionState: Hive Session ID = 32336280-d41f-4a0a-b3d7-b39399f2bbab 2025-10-15 20:24:07,617 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:07,656 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/32336280-d41f-4a0a-b3d7-b39399f2bbab 2025-10-15 20:24:07,670 INFO session.SessionState: Created local directory: /opt/installs/hive3.1.2/iotmp/root/32336280-d41f-4a0a-b3d7-b39399f2bbab 2025-10-15 20:24:07,681 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/32336280-d41f-4a0a-b3d7-b39399f2bbab/_tmp_space.db 2025-10-15 20:24:07,865 INFO sqlstd.SQLStdHiveAccessController: Created SQLStdHiveAccessController for session context : HiveAuthzSessionContext [sessionString=a3156ff3-c06b-4333-9294-e6fd634a4b9d, clientType=HIVECLI] 2025-10-15 20:24:07,891 WARN session.SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory. 2025-10-15 20:24:07,944 INFO metastore.HiveMetaStoreClient: Mestastore configuration metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook 2025-10-15 20:24:07,974 INFO metastore.HiveMetaStoreClient: Closed a connection to metastore, current connections: 0 2025-10-15 20:24:07,977 ERROR utils.MetaStoreUtils: Got exception: org.apache.thrift.transport.TTransportException Cannot write to null outputStream org.apache.thrift.transport.TTransportException: Cannot write to null outputStream at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142) at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:178) at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:106) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:70) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_tables_by_type(ThriftHiveMetastore.java:1913) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_tables_by_type(ThriftHiveMetastore.java:1903) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:1676) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:1665) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) at com.sun.proxy.$Proxy41.getTables(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773) at com.sun.proxy.$Proxy41.getTables(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByType(Hive.java:1310) at org.apache.hadoop.hive.ql.metadata.Hive.getTableObjects(Hive.java:1222) at org.apache.hadoop.hive.ql.metadata.Hive.getAllMaterializedViewObjects(Hive.java:1217) at org.apache.hadoop.hive.ql.metadata.HiveMaterializedViewsRegistry$Loader.run(HiveMaterializedViewsRegistry.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2025-10-15 20:24:07,983 ERROR utils.MetaStoreUtils: Converting exception to MetaException 2025-10-15 20:24:07,980 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://caiji:9083 2025-10-15 20:24:07,984 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 1 2025-10-15 20:24:07,992 INFO metastore.HiveMetaStoreClient: Connected to metastore. 2025-10-15 20:24:07,992 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=root (auth:SIMPLE) retries=1 delay=1 lifetime=0 2025-10-15 20:24:08,002 WARN metastore.RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect (1 of 1) after 1s. getTables MetaException(message:Got exception: org.apache.thrift.transport.TTransportException Cannot write to null outputStream) at org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:168) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:1667) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) at com.sun.proxy.$Proxy41.getTables(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773) at com.sun.proxy.$Proxy41.getTables(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByType(Hive.java:1310) at org.apache.hadoop.hive.ql.metadata.Hive.getTableObjects(Hive.java:1222) at org.apache.hadoop.hive.ql.metadata.Hive.getAllMaterializedViewObjects(Hive.java:1217) at org.apache.hadoop.hive.ql.metadata.HiveMaterializedViewsRegistry$Loader.run(HiveMaterializedViewsRegistry.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2025-10-15 20:24:08,212 INFO SessionState: Added [/opt/installs/hive3.1.2/iotmp/a3156ff3-c06b-4333-9294-e6fd634a4b9d_resources/json-udf-1.3.8-jar-with-dependencies.jar] to class path 2025-10-15 20:24:08,212 INFO SessionState: Added resources: [hdfs:///common/lib/json-udf-1.3.8-jar-with-dependencies.jar] 2025-10-15 20:24:08,213 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:08,213 INFO session.SessionState: Resetting thread name to main 2025-10-15 20:24:08,213 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:08,213 INFO session.SessionState: Updating thread name to a3156ff3-c06b-4333-9294-e6fd634a4b9d main 2025-10-15 20:24:08,258 INFO SessionState: Added [/opt/installs/hive3.1.2/iotmp/a3156ff3-c06b-4333-9294-e6fd634a4b9d_resources/json-serde-1.3.8-jar-with-dependencies.jar] to class path 2025-10-15 20:24:08,258 INFO SessionState: Added resources: [hdfs:///common/lib/json-serde-1.3.8-jar-with-dependencies.jar] 2025-10-15 20:24:08,258 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:08,258 INFO session.SessionState: Resetting thread name to main 2025-10-15 20:24:08,258 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:08,258 INFO session.SessionState: Updating thread name to a3156ff3-c06b-4333-9294-e6fd634a4b9d main 2025-10-15 20:24:08,297 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:08,297 INFO session.SessionState: Resetting thread name to main 2025-10-15 20:24:08,298 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:08,298 INFO session.SessionState: Updating thread name to a3156ff3-c06b-4333-9294-e6fd634a4b9d main 2025-10-15 20:24:08,299 INFO SessionState: Added [/opt/installs/hive3.1.2/iotmp/a3156ff3-c06b-4333-9294-e6fd634a4b9d_resources/json-udf-1.3.8-jar-with-dependencies.jar] to class path 2025-10-15 20:24:08,300 INFO SessionState: Added resources: [hdfs:///common/lib/json-udf-1.3.8-jar-with-dependencies.jar] 2025-10-15 20:24:08,300 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:08,300 INFO session.SessionState: Resetting thread name to main 2025-10-15 20:24:08,300 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:08,300 INFO session.SessionState: Updating thread name to a3156ff3-c06b-4333-9294-e6fd634a4b9d main 2025-10-15 20:24:08,301 INFO SessionState: Added [/opt/installs/hive3.1.2/iotmp/a3156ff3-c06b-4333-9294-e6fd634a4b9d_resources/json-serde-1.3.8-jar-with-dependencies.jar] to class path 2025-10-15 20:24:08,301 INFO SessionState: Added resources: [hdfs:///common/lib/json-serde-1.3.8-jar-with-dependencies.jar] 2025-10-15 20:24:08,301 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:08,302 INFO session.SessionState: Resetting thread name to main 2025-10-15 20:24:08,303 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:08,303 INFO session.SessionState: Updating thread name to a3156ff3-c06b-4333-9294-e6fd634a4b9d main 2025-10-15 20:24:08,845 INFO ql.Driver: Compiling command(queryId=root_20251015202408_6d0ebb16-1bd2-4867-9b4c-c4ef8da002a7): CREATE TABLE IF NOT EXISTS `dim_nshop`.`dim_pub_area` ( `region_code` STRING, `region_code_desc` STRING, `region_city` STRING, `region_city_desc` STRING, `region_province` STRING, `region_province_desc` STRING) COMMENT 'Imported by sqoop on 2025/10/15 20:23:53' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE 2025-10-15 20:24:09,011 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient trying reconnect as root (auth:SIMPLE) 2025-10-15 20:24:09,015 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://caiji:9083 2025-10-15 20:24:09,018 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 2 2025-10-15 20:24:09,022 INFO metastore.HiveMetaStoreClient: Connected to metastore. 2025-10-15 20:24:09,072 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://caiji:9083 2025-10-15 20:24:09,072 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 3 2025-10-15 20:24:09,076 INFO metastore.HiveMetaStoreClient: Connected to metastore. 2025-10-15 20:24:09,076 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=root (auth:SIMPLE) retries=1 delay=1 lifetime=0 2025-10-15 20:24:09,358 INFO metadata.HiveMaterializedViewsRegistry: Materialized views registry has been initialized 2025-10-15 20:24:13,737 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 2025-10-15 20:24:13,760 INFO parse.CalcitePlanner: Starting Semantic Analysis 2025-10-15 20:24:13,825 INFO parse.CalcitePlanner: Creating table dim_nshop.dim_pub_area position=27 2025-10-15 20:24:14,225 INFO ql.Driver: Semantic Analysis Completed (retrial = false) 2025-10-15 20:24:14,246 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null) 2025-10-15 20:24:14,319 INFO ql.Driver: Completed compiling command(queryId=root_20251015202408_6d0ebb16-1bd2-4867-9b4c-c4ef8da002a7); Time taken: 5.741 seconds 2025-10-15 20:24:14,320 INFO reexec.ReExecDriver: Execution #1 of query 2025-10-15 20:24:14,329 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 2025-10-15 20:24:14,330 INFO ql.Driver: Executing command(queryId=root_20251015202408_6d0ebb16-1bd2-4867-9b4c-c4ef8da002a7): CREATE TABLE IF NOT EXISTS `dim_nshop`.`dim_pub_area` ( `region_code` STRING, `region_code_desc` STRING, `region_city` STRING, `region_city_desc` STRING, `region_province` STRING, `region_province_desc` STRING) COMMENT 'Imported by sqoop on 2025/10/15 20:23:53' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE 2025-10-15 20:24:14,413 INFO ql.Driver: Completed executing command(queryId=root_20251015202408_6d0ebb16-1bd2-4867-9b4c-c4ef8da002a7); Time taken: 0.084 seconds OK 2025-10-15 20:24:14,413 INFO ql.Driver: OK 2025-10-15 20:24:14,414 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager Time taken: 5.849 seconds 2025-10-15 20:24:14,417 INFO CliDriver: Time taken: 5.849 seconds 2025-10-15 20:24:14,417 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:14,417 INFO session.SessionState: Resetting thread name to main 2025-10-15 20:24:14,418 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:14,418 INFO session.SessionState: Updating thread name to a3156ff3-c06b-4333-9294-e6fd634a4b9d main 2025-10-15 20:24:14,424 INFO ql.Driver: Compiling command(queryId=root_20251015202414_6fef43ef-ccc7-4ebf-89b4-ef659209e529): LOAD DATA INPATH 'hdfs://caiji:9820/data/nshop/ods/dim_pub_area' INTO TABLE `dim_nshop`.`dim_pub_area` 2025-10-15 20:24:14,460 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 2025-10-15 20:24:15,264 INFO ql.Driver: Semantic Analysis Completed (retrial = false) 2025-10-15 20:24:15,265 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null) 2025-10-15 20:24:15,265 INFO ql.Driver: Completed compiling command(queryId=root_20251015202414_6fef43ef-ccc7-4ebf-89b4-ef659209e529); Time taken: 0.841 seconds 2025-10-15 20:24:15,265 INFO reexec.ReExecDriver: Execution #1 of query 2025-10-15 20:24:15,266 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 2025-10-15 20:24:15,266 INFO ql.Driver: Executing command(queryId=root_20251015202414_6fef43ef-ccc7-4ebf-89b4-ef659209e529): LOAD DATA INPATH 'hdfs://caiji:9820/data/nshop/ods/dim_pub_area' INTO TABLE `dim_nshop`.`dim_pub_area` 2025-10-15 20:24:15,301 INFO ql.Driver: Starting task [Stage-0:MOVE] in serial mode 2025-10-15 20:24:15,306 INFO metastore.HiveMetaStoreClient: Closed a connection to metastore, current connections: 2 Loading data to table dim_nshop.dim_pub_area 2025-10-15 20:24:15,306 INFO exec.Task: Loading data to table dim_nshop.dim_pub_area from hdfs://caiji:9820/data/nshop/ods/dim_pub_area 2025-10-15 20:24:15,313 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://caiji:9083 2025-10-15 20:24:15,313 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 3 2025-10-15 20:24:15,320 INFO metastore.HiveMetaStoreClient: Connected to metastore. 2025-10-15 20:24:15,320 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=root (auth:SIMPLE) retries=1 delay=1 lifetime=0 2025-10-15 20:24:15,975 INFO ql.Driver: Starting task [Stage-1:STATS] in serial mode 2025-10-15 20:24:15,976 INFO metastore.HiveMetaStoreClient: Closed a connection to metastore, current connections: 2 2025-10-15 20:24:15,976 INFO stats.BasicStatsTask: Executing stats task 2025-10-15 20:24:15,999 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://caiji:9083 2025-10-15 20:24:15,999 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 3 2025-10-15 20:24:16,004 INFO metastore.HiveMetaStoreClient: Connected to metastore. 2025-10-15 20:24:16,004 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=root (auth:SIMPLE) retries=1 delay=1 lifetime=0 2025-10-15 20:24:16,948 INFO stats.BasicStatsTask: Table dim_nshop.dim_pub_area stats: [numFiles=2, numRows=0, totalSize=66444, rawDataSize=0] 2025-10-15 20:24:16,949 INFO ql.Driver: Completed executing command(queryId=root_20251015202414_6fef43ef-ccc7-4ebf-89b4-ef659209e529); Time taken: 1.682 seconds OK 2025-10-15 20:24:16,949 INFO ql.Driver: OK 2025-10-15 20:24:16,949 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager Time taken: 2.525 seconds 2025-10-15 20:24:16,949 INFO CliDriver: Time taken: 2.525 seconds 2025-10-15 20:24:16,949 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:16,949 INFO session.SessionState: Resetting thread name to main 2025-10-15 20:24:16,950 INFO conf.HiveConf: Using the default value passed in for log id: a3156ff3-c06b-4333-9294-e6fd634a4b9d 2025-10-15 20:24:16,980 INFO session.SessionState: Deleted directory: /tmp/hive/root/a3156ff3-c06b-4333-9294-e6fd634a4b9d on fs with scheme hdfs 2025-10-15 20:24:16,982 INFO session.SessionState: Deleted directory: /opt/installs/hive3.1.2/iotmp/root/a3156ff3-c06b-4333-9294-e6fd634a4b9d on fs with scheme file 2025-10-15 20:24:16,983 INFO metastore.HiveMetaStoreClient: Closed a connection to metastore, current connections: 2 2025-10-15 20:24:16,984 INFO hive.HiveImport: Hive import complete. 2025-10-15 20:24:16,990 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory. [root@caiji ~]# Hive Session ID = a9894d69-9202-4b58-9ca0-6eef301aed2c Hive Session ID = ae9cf9ab-f39f-4427-ac3a-5814406e64aa Hive Session ID = 29df4bf0-a832-4031-b833-29dd1fafb9c8 Hive Session ID = b3f7b733-9fc0-43f1-9eed-dda7772f5f80 Hive Session ID = be9341af-4b2e-4f80-b38a-366118d7b883 Hive Session ID = 6b18de3f-9c17-4284-b156-e620d002430c Hive Session ID = 45b5e4e2-c4d5-4f4c-90e3-ac3dded07d75 Hive Session ID = a0d8dc7d-cf69-4027-9c73-72585f89b631 Hive Session ID = c172f183-771d-4192-b320-1c076d8e1bcc Hive Session ID = 7320aca4-1bba-4271-b1d8-847b5f028c85 Hive Session ID = d6498fc1-0e11-4cd7-925b-4d7f4dd5a67c Hive Session ID = f9f74c84-604d-450e-8181-b8c3484aad10
最新发布
10-16
2. 从类路径中移除不需要的 SLF4J 绑定 JAR 文件。例如，如果要移除 `slf4j-log4j12-1.7.10.jar`，可以使用以下命令： ```sh rm /usr/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar ``` ### 解决 Java ...
centos7+hadoop3.2.2+mysql5.7.33安装hive3.1.2
m0_37813354的博客
05-29 1853
1 下载hive安装包 http://ftp.twaren.net/Unix/Web/apache/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz 2 上传解压 [liucf@node1 softfile]$ tar -zxf apache-hive-3.1.2-bin.tar.gz -C ../soft 修改一下解压后的文件目录名，是自己看着舒服点，可以不做 [liucf@node1 soft]$ mv apache-hive-3.1.2-bin a
[root@node ~]# mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8 Server version: 8.0.42 MySQL Community Server - GPL Copyright (c) 2000, 2025, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> CREATE DATABASE weblog_db; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'CREATE DATABASE weblog_db' at line 1 mysql> CREATE DATABASE weblog_db; ERROR 1007 (HY000): Can't create database 'weblog_db'; database exists mysql> USE weblog_db; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> DROP DATABASE IF EXISTS weblog_db; Query OK, 2 rows affected (0.02 sec) mysql> CREATE DATABASE weblog_db; Query OK, 1 row affected (0.01 sec) mysql> USE weblog_db; Database changed mysql> CREATE TABLE page_visits ( -> page VARCHAR(255) , -> visits BIGINT -> ); ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'TABLE page_visits ( page VARCHAR(255) , visits BIGINT )' at line 1 mysql> CREATE TABLE page_visits ( -> page VARCHAR(255), -> visits BIGINT -> ); Query OK, 0 rows affected (0.02 sec) mysql> SHOW TABLES; +---------------------+ | Tables_in_weblog_db | +---------------------+ | page_visits | +---------------------+ 1 row in set (0.00 sec) mysql> ^C mysql> q -> quit -> exit -> ^C mysql> ^C mysql> ^C mysql> ^DBye [root@node ~]# hive SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Hive Session ID = 7bb79582-cc2b-49b6-abc7-020dcdc46542 Logging initialized using configuration in jar:file:/home/hive-3.1.3/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Hive Session ID = 15d9da52-e18e-40b2-a80f-e76eda81df4c hive> DESCRIBE FORMATTED page_visits; OK # col_name data_type comment page string visits bigint # Detailed Table Information Database: default OwnerType: USER Owner: root CreateTime: Tue Jul 08 01:43:42 CST 2025 LastAccessTime: UNKNOWN Retention: 0 Location: hdfs://node:9000/hive/warehouse/page_visits Table Type: MANAGED_TABLE Table Parameters: COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"} bucketing_version 2 numFiles 1 numRows 4 rawDataSize 56 totalSize 60 transient_lastDdlTime 1751910222 # Storage Information SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe InputFormat: org.apache.hadoop.mapred.TextInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Compressed: No Num Buckets: -1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: serialization.format 1 Time taken: 0.785 seconds, Fetched: 32 row(s) hive> [root@node ~]# [root@node ~]# sqoop export \ > --connect jdbc:mysql://localhost/weblog_db \ > --username root \ > --password Aa@123456 \ > --table page_visits \ > --export-dir hdfs://node:9000/hive/warehouse/page_visits \ > --input-fields-terminated-by '\001' \ > --num-mappers 1 Warning: /home/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /home/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 2025-07-08 15:28:12,550 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 2025-07-08 15:28:12,587 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 2025-07-08 15:28:12,704 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 2025-07-08 15:28:12,708 INFO tool.CodeGenTool: Beginning code generation Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 2025-07-08 15:28:13,225 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `page_visits` AS t LIMIT 1 2025-07-08 15:28:13,266 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `page_visits` AS t LIMIT 1 2025-07-08 15:28:13,280 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop3.3 Note: /tmp/sqoop-root/compile/363869e21c2078b9742685122c43a3cc/page_visits.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 2025-07-08 15:28:16,377 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/363869e21c2078b9742685122c43a3cc/page_visits.jar 2025-07-08 15:28:16,391 INFO mapreduce.ExportJobBase: Beginning export of page_visits 2025-07-08 15:28:16,391 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2025-07-08 15:28:16,484 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 2025-07-08 15:28:17,339 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 2025-07-08 15:28:17,342 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 2025-07-08 15:28:17,343 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 2025-07-08 15:28:17,555 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at node/192.168.196.122:8032 2025-07-08 15:28:17,782 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1751959003014_0001 2025-07-08 15:28:26,026 INFO input.FileInputFormat: Total input files to process : 1 2025-07-08 15:28:26,029 INFO input.FileInputFormat: Total input files to process : 1 2025-07-08 15:28:26,495 INFO mapreduce.JobSubmitter: number of splits:1 2025-07-08 15:28:26,528 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 2025-07-08 15:28:26,619 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1751959003014_0001 2025-07-08 15:28:26,620 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2025-07-08 15:28:26,805 INFO conf.Configuration: resource-types.xml not found 2025-07-08 15:28:26,805 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2025-07-08 15:28:27,226 INFO impl.YarnClientImpl: Submitted application application_1751959003014_0001 2025-07-08 15:28:27,264 INFO mapreduce.Job: The url to track the job: http://node:8088/proxy/application_1751959003014_0001/ 2025-07-08 15:28:27,264 INFO mapreduce.Job: Running job: job_1751959003014_0001 2025-07-08 15:28:34,334 INFO mapreduce.Job: Job job_1751959003014_0001 running in uber mode : false 2025-07-08 15:28:34,335 INFO mapreduce.Job: map 0% reduce 0% 2025-07-08 15:28:38,374 INFO mapreduce.Job: map 100% reduce 0% 2025-07-08 15:28:38,381 INFO mapreduce.Job: Job job_1751959003014_0001 failed with state FAILED due to: Task failed task_1751959003014_0001_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 2025-07-08 15:28:38,448 INFO mapreduce.Job: Counters: 8 Job Counters Failed map tasks=1 Launched map tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=2061 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=2061 Total vcore-milliseconds taken by all map tasks=2061 Total megabyte-milliseconds taken by all map tasks=2110464 2025-07-08 15:28:38,456 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 2025-07-08 15:28:38,457 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 21.1033 seconds (0 bytes/sec) 2025-07-08 15:28:38,462 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 2025-07-08 15:28:38,462 INFO mapreduce.ExportJobBase: Exported 0 records. 2025-07-08 15:28:38,462 ERROR mapreduce.ExportJobBase: Export job failed! 2025-07-08 15:28:38,463 ERROR tool.ExportTool: Error during export: Export job failed! at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:445) at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) [root@node ~]# sqoop export \ > --connect jdbc:mysql://localhost/weblog_db \ > --username root \ > --password Aa@123456 \ > --table page_visits \ > --export-dir hdfs://node:9000/hive/warehouse/page_visits \ > --input-fields-terminated-by ',' \ > --num-mappers 1 Warning: /home/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /home/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 2025-07-08 15:30:31,174 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 2025-07-08 15:30:31,218 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 2025-07-08 15:30:31,333 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 2025-07-08 15:30:31,336 INFO tool.CodeGenTool: Beginning code generation Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 2025-07-08 15:30:31,771 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `page_visits` AS t LIMIT 1 2025-07-08 15:30:31,814 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `page_visits` AS t LIMIT 1 2025-07-08 15:30:31,821 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop3.3 Note: /tmp/sqoop-root/compile/ab00e36d1f5084a0f7d522b4e9a975e5/page_visits.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 2025-07-08 15:30:33,116 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/ab00e36d1f5084a0f7d522b4e9a975e5/page_visits.jar 2025-07-08 15:30:33,129 INFO mapreduce.ExportJobBase: Beginning export of page_visits 2025-07-08 15:30:33,129 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2025-07-08 15:30:33,212 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 2025-07-08 15:30:33,877 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 2025-07-08 15:30:33,880 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 2025-07-08 15:30:33,880 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 2025-07-08 15:30:34,097 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at node/192.168.196.122:8032 2025-07-08 15:30:34,310 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1751959003014_0002 2025-07-08 15:30:39,127 INFO input.FileInputFormat: Total input files to process : 1 2025-07-08 15:30:39,131 INFO input.FileInputFormat: Total input files to process : 1 2025-07-08 15:30:39,995 INFO mapreduce.JobSubmitter: number of splits:1 2025-07-08 15:30:40,022 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 2025-07-08 15:30:40,532 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1751959003014_0002 2025-07-08 15:30:40,532 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2025-07-08 15:30:40,689 INFO conf.Configuration: resource-types.xml not found 2025-07-08 15:30:40,689 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2025-07-08 15:30:40,746 INFO impl.YarnClientImpl: Submitted application application_1751959003014_0002 2025-07-08 15:30:40,783 INFO mapreduce.Job: The url to track the job: http://node:8088/proxy/application_1751959003014_0002/ 2025-07-08 15:30:40,784 INFO mapreduce.Job: Running job: job_1751959003014_0002 2025-07-08 15:30:46,847 INFO mapreduce.Job: Job job_1751959003014_0002 running in uber mode : false 2025-07-08 15:30:46,848 INFO mapreduce.Job: map 0% reduce 0% 2025-07-08 15:30:50,893 INFO mapreduce.Job: map 100% reduce 0% 2025-07-08 15:30:51,905 INFO mapreduce.Job: Job job_1751959003014_0002 failed with state FAILED due to: Task failed task_1751959003014_0002_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 2025-07-08 15:30:51,973 INFO mapreduce.Job: Counters: 8 Job Counters Failed map tasks=1 Launched map tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=2058 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=2058 Total vcore-milliseconds taken by all map tasks=2058 Total megabyte-milliseconds taken by all map tasks=2107392 2025-07-08 15:30:51,979 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 2025-07-08 15:30:51,980 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 18.0828 seconds (0 bytes/sec) 2025-07-08 15:30:51,983 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 2025-07-08 15:30:51,983 INFO mapreduce.ExportJobBase: Exported 0 records. 2025-07-08 15:30:51,983 ERROR mapreduce.ExportJobBase: Export job failed! 2025-07-08 15:30:51,984 ERROR tool.ExportTool: Error during export: Export job failed! at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:445) at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) [root@node ~]# 6.2 Sqoop导出数据 6.2.1从Hive将数据导出到MySQL 6.2.2sqoop导出格式 6.2.3导出page_visits表 6.2.4导出到ip_visits表 6.3验证导出数据 6.3.1登录MySQL 6.3.2执行查询
07-09
但是，在导出作业中，Sqoop首先会连接MySQL获取元数据（从日志中可以看到它执行了`SELECT t.* FROM page_visits AS t LIMIT 1`），这说明连接MySQL是成功的。 因此，问题可能出现在读取HDFS文件并解析的时候。 ...
Sqoop应用_导入Hive
weixin_45017098的博客
06-20 1782
--incremental lastmodified --merge-key的作用：修改过的数据和新增的数据（前提是满足last-value的条件）都会导入进来，并且重复的数据（不需要满足last-value的条件）都会进行合并。如果之前的数据有修改的话可以使用–incremental lastmodified --merge-key进行数据合并执行修改的SQL。现在我们已经实现了 hive的数据导入方式,那么我们怎么实现hive的增量数据导入呢?结果：id=3的数据成功导入。1、append方式。
sqoop1安装在hadoop2.6.0上不能使用import命令的问题
Chunhong Song的专栏
08-03 1584
本人是菜鸟一枚，之前自己就在dell的台式机上安装了hadoop2.6.0的伪分布式，一直认为hadoop高深非我等常人可以明白理解的，怀揣着不求机理、问题清楚明了，只求能折腾出我想要的结果的心游走在Hadoop的边缘，所谓擦边球。但最近发现很多功能试了又试就是达不到预想的结果。然后网上搜了n多解决方案，依然不能解决。看了一个帖子，评论上说回归logs，如果连logs都看不明白，那你只能上个棒槌了...
sqoop工具常用Hive操作
nisjlvhudy的专栏
11-05 1539
 1、查看sqoop命令说明 [spark@Master data_dir]$ sqoop help usage: sqoop COMMAND [ARGS] Available commands: codegen Generate code to interact with database records create-hive-table Impo
jdbc连接Hive
qq_34792920的博客
01-02 4453
jdbc连接Hive 1.使用sqoop将stu表导入到hive中 ​ 数据库表位于hadoop102上的test数据库 ​ bin/sqoop import \ --connect jdbc:mysql://hadoop102:3306/test \ --username root \ --password 000000 \ --table stu \ --num-mappers 1 \ --hive-import \ --fields-terminated-by "\t" \ --hive-overwr
记录一次hiveserver2卡死(假死)问题
weixin_41956627的博客
10-14 1178
由于执行的sql太长导致hive卡死的问题记录
一百一十五、（未解决）Sqoop报错：Sqoop连clickhouse：No manager for connect string: jdbc:clickhouse://hurys22:8123
tiantang2renjian的博客
05-16 578
Sqoop报错：Sqoop连clickhouse：No manager for connect string: jdbc:clickhouse://hurys22:8123
Sqoop数据导入到Hive表的最佳实践
日常分享数据分析开发、编程语言内容
01-22 2220
Sqoop是一个开源工具，用于在Hadoop生态系统中的数据存储（如HDFS）和关系型数据库之间传输数据。它可以帮助数据工程师和分析师轻松地将结构化数据从关系型数据库导入到Hadoop集群中，以供进一步的数据处理和分析。将数据从关系型数据库导入到Hive表是大数据分析中的关键步骤之一。本文提供了Sqoop数据导入到Hive表的最佳实践，包括详细的步骤、示例代码和最佳建议。希望这些示例代码和详细内容有助于大家更好地理解和实施数据导入操作。
使用sqoop导入数据到hive问题总结
09-27 601
解决过程：网上搜索很多的原因都说是因为mapreduce内存或者是datanode内存不足，各种配置都没解决，最后检查hive建表语句发现字段对不上，还是太粗心。原因：hive表字段指定了分区，但导入的时候没有指定分区。解决：修改hive表字段或者指定分区。原因：mysql驱动包搞错了。解决：更换mysql驱动包。
Sqoop导入数据库数据到Hive的两种方式
gxd520的专栏
08-30 5295
通过Sqoop将关系型数据库数据到Hive有两种方式，一种是Sqoop API，一种是使用HCatalog API。两种方式略有不同，各有优缺点，这里主要介绍两者的区别与共同点。 Sqoop方式 Sqoop方式基本的命令如下： sqoop import \ --hive-import \ --connect 'jdbc:mysql://localhost:3306/test' \ --userna...
sqoop工具配合sqljdbc4-4.jar导入导出数据指南
- 使用Sqoop的命令行工具执行数据导入或导出操作。导入操作通常使用import命令，导出操作通常使用export命令。 - 在命令行中指定JDBC连接字符串、目标Hive表或SQL Server表、字段分隔符、表分隔符等参数，以及可能...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值