连接Oracle与Hadoop(3) 使用OLH加载Hbase到Oracle

最新推荐文章于 2022-09-10 13:28:37 发布
最新推荐文章于 2022-09-10 13:28:37 发布
阅读量107 收藏
点赞数
文章标签： java 数据库 大数据
原文链接：http://www.cnblogs.com/panwenyu/p/5058590.html
版权

OLH是Oracle Loader for Hadoop的缩写，Oracle的大数据连接器（BDC）的一个组件，可将多种数据格式从HDFS上加载到Oracle数据库库中。

 

本文在同一台服务器上模拟oracle数据库与hadoop集群，实验目标：使用OLH从Hadoop端的Hbase表加载数据到Oracle表中。

对于加载Hbase表场景，需要通过Hive创建一个基于Hbase的映射表，再通过Hive接口导入到Oracle表中。

 

Oracle端：

服务器

系统用户

安装软件

软件安装路径

Server1

oracle

Oracle Database 12.1.0.2

/u01/app/oracle/product/12.1.0/dbhome_1

 

Hadoop集群端：

服务器

系统用户

安装软件

软件安装路径

Server1

hadoop

Hadoop 2.6.2

/home/hadoop/hadoop-2.6.2

Hive 1.1.1

/home/hadoop/hive-1.1.1

Hbase 1.1.2

/home/hadoop/hbase-1.1.2

jdk1.8.0_65

/home/hadoop/jdk1.8.0_65

OLH 3.5.0

/home/hadoop/oraloader-3.5.0-h2

 

  • 部署Hadoop/Hive/Hbase/OLH软件

将Hadoop/Hive/Hbase/OLH软件解压到相应目录

  1. [hadoop@server1 ~]$ tree -L 1
  2. ├── hadoop-2.6.2
  3. ├── hbase-1.1.2
  4. ├── hive-1.1.1
  5. ├── jdk1.8.0_65
  6. ├── oraloader-3.5.0-h2

 

  • 配置Hadoop/Hive/Hbase/OLH环境变量
  1. export JAVA_HOME=/home/hadoop/jdk1.8.0_65
  2.  
  3. export HADOOP_USER_NAME=hadoop
  4. export HADOOP_YARN_USER=hadoop
  5. export HADOOP_HOME=/home/hadoop/hadoop-2.6.2
  6. export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
  7. export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
  8. export HADOOP_LIBEXEC_DIR=${HADOOP_HOME}/libexec
  9. export HADOOP_COMMON_HOME=${HADOOP_HOME}
  10. export HADOOP_HDFS_HOME=${HADOOP_HOME}
  11. export HADOOP_MAPRED_HOME=${HADOOP_HOME}
  12. export HADOOP_YARN_HOME=${HADOOP_HOME}
  13. export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
  14. export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
  15.  
  16. export HIVE_HOME=/home/hadoop/hive-1.1.1
  17. export HIVE_CONF_DIR=${HIVE_HOME}/conf
  18.  
  19. export HBASE_HOME=/home/hadoop/hbase-1.1.2
  20. export HBASE_CONF_DIR=/home/hadoop/hbase-1.1.2/conf
  21.  
  22. export OLH_HOME=/home/hadoop/oraloader-3.5.0-h2
  23.  
  24. export HADOOP_CLASSPATH=/usr/share/java/mysql-connector-java.jar
  25. export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*:$HIVE_CONF_DIR:$HBASE_HOME/lib/*:$HBASE_CONF_DIR
  26. export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$OLH_HOME/jlib/*
  27.  
  28. export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$HBASE_HOME/bin:$PATH

 

  • 配置Hadoop/Hive/Hbase软件

core-site.xml

  1. <configuration>
  2. <property>
  3.         <name>fs.defaultFS</name>
  4.         <value>hdfs://server1:8020</value>
  5. </property>
  6. <property>
  7.         <name>fs.checkpoint.period</name>
  8.         <value>3600</value>
  9. </property>
  10. <property>
  11.         <name>fs.checkpoint.size</name>
  12.         <value>67108864</value>
  13. </property>
  14. <property>
  15.         <name>hadoop.proxyuser.hadoop.groups</name>
  16.         <value>*</value>
  17. </property>
  18. <property>
  19.         <name>hadoop.proxyuser.hadoop.hosts</name>
  20.         <value>*</value>
  21. </property>
  22. </configuration>

 

hdfs-site.xml

  1. <configuration>
  2. <property>
  3.         <name>hadoop.tmp.dir</name>
  4.         <value>file:///home/hadoop</value>
  5. </property>
  6. <property>
  7.         <name>dfs.namenode.name.dir</name>
  8.         <value>file:///home/hadoop/dfs/nn</value>
  9. </property>
  10. <property>
  11.         <name>dfs.datanode.data.dir</name>
  12.         <value>file:///home/hadoop/dfs/dn</value>
  13. </property>
  14. <property>
  15.         <name>dfs.namenode.checkpoint.dir</name>
  16.         <value>file:///home/hadoop/dfs/sn</value>
  17. </property>
  18. <property>
  19.         <name>dfs.replication</name>
  20.         <value>1</value>
  21. </property>
  22. <property>
  23.         <name>dfs.permissions.superusergroup</name>
  24.         <value>supergroup</value>
  25. </property>
  26. <property>
  27.         <name>dfs.namenode.http-address</name>
  28.         <value>server1:50070</value>
  29. </property>
  30. <property>
  31.         <name>dfs.namenode.secondary.http-address</name>
  32.         <value>server1:50090</value>
  33. </property>
  34. <property>
  35.         <name>dfs.webhdfs.enabled</name>
  36.         <value>true</value>
  37. </property>
  38. </configuration>

 

yarn-site.xml

  1. <configuration>
  2. <property>
  3.         <name>yarn.resourcemanager.scheduler.address</name>
  4.         <value>server1:8030</value>
  5. </property>
  6. <property>
  7.         <name>yarn.resourcemanager.resource-tracker.address</name>
  8.         <value>server1:8031</value>
  9. </property>
  10. <property>
  11.         <name>yarn.resourcemanager.address</name>
  12.         <value>server1:8032</value>
  13. </property>
  14. <property>
  15.         <name>yarn.resourcemanager.admin.address</name>
  16.         <value>server1:8033</value>
  17. </property>
  18. <property>
  19.         <name>yarn.resourcemanager.webapp.address</name>
  20.         <value>server1:8088</value>
  21. </property>
  22. <property>
  23.         <name>yarn.nodemanager.local-dirs</name>
  24.         <value>file:///home/hadoop/yarn/local</value>
  25. </property>
  26. <property>
  27.         <name>yarn.nodemanager.log-dirs</name>
  28.         <value>file:///home/hadoop/yarn/logs</value>
  29. </property>
  30. <property>
  31.         <name>yarn.log-aggregation-enable</name>
  32.         <value>true</value>
  33. </property>
  34. <property>
  35.         <name>yarn.nodemanager.remote-app-log-dir</name>
  36.         <value>/yarn/apps</value>
  37. </property>
  38. <property>
  39.         <name>yarn.app.mapreduce.am.staging-dir</name>
  40.         <value>/user</value>
  41. </property>
  42. <property>
  43.         <name>yarn.nodemanager.aux-services</name>
  44.         <value>mapreduce_shuffle</value>
  45. </property>
  46. <property>
  47.         <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
  48.         <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  49. </property>
  50. <property>
  51.         <name>yarn.log.server.url</name>
  52.         <value>http://server1:19888/jobhistory/logs/</value>
  53. </property>
  54. </configuration>

 

  • 配置Hive

hive-site.xml

  1. <configuration>
  2. <property>
  3.         <name>javax.jdo.option.ConnectionURL</name>
  4.         <value>jdbc:mysql://server1:3306/metastore?createDatabaseIfNotExist=true</value>
  5. </property>
  6. <property>
  7.         <name>javax.jdo.option.ConnectionDriverName</name>
  8.         <value>com.mysql.jdbc.Driver</value>
  9. </property>
  10. <property>
  11.         <name>javax.jdo.option.ConnectionUserName</name>
  12.         <value>hive</value>
  13. </property>
  14. <property>
  15.         <name>javax.jdo.option.ConnectionPassword</name>
  16.         <value>123456</value>
  17. </property>
  18. <property>
  19.         <name>hbase.master</name>
  20.         <value>server1:16000</value>
  21. </property>
  22. <property>
  23.         <name>hive.cli.print.current.db</name>
  24.         <value>true</value>
  25. </property>
  26. </configuration>

 

  • 配置Hbase

Hbase-site.xml

  1. <configuration>
  2. <property>
  3.         <name>hbase.rootdir</name>
  4.         <value>hdfs://server1/hbase</value>
  5. </property>
  6. <property>
  7.         <name>hbase.cluster.distributed</name>
  8.         <value>true</value>
  9. </property>
  10. <property>
  11.         <name>hbase.master.port</name>
  12.         <value>16000</value>
  13. </property>
  14. </configuration>

 

  • 创建一张Hbase表
  1. create 'WLSSERVER' , 'LOG'
  2. put 'WLSSERVER', 'row1', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:16-PM-PDT'
  3. put 'WLSSERVER', 'row1', 'LOG:CATEGORY', 'Notice'
  4. put 'WLSSERVER', 'row1', 'LOG:TYPE', 'WebLogicServer'
  5. put 'WLSSERVER', 'row1', 'LOG:SERVERNAME', 'AdminServer'
  6. put 'WLSSERVER', 'row1', 'LOG:CODE', 'BEA-000365'
  7. put 'WLSSERVER', 'row1', 'LOG:MSG', 'Server state changed to STANDBY'
  8. put 'WLSSERVER', 'row2', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:17-PM-PDT'
  9. put 'WLSSERVER', 'row2', 'LOG:CATEGORY', 'Notice'
  10. put 'WLSSERVER', 'row2', 'LOG:TYPE', 'WebLogicServer'
  11. put 'WLSSERVER', 'row2', 'LOG:SERVERNAME', 'AdminServer'
  12. put 'WLSSERVER', 'row2', 'LOG:CODE', 'BEA-000365'
  13. put 'WLSSERVER', 'row2', 'LOG:MSG', 'Server state changed to STARTING'
  14. put 'WLSSERVER', 'row3', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:18-PM-PDT'
  15. put 'WLSSERVER', 'row3', 'LOG:CATEGORY', 'Notice'
  16. put 'WLSSERVER', 'row3', 'LOG:TYPE', 'WebLogicServer'
  17. put 'WLSSERVER', 'row3', 'LOG:SERVERNAME', 'AdminServer'
  18. put 'WLSSERVER', 'row3', 'LOG:CODE', 'BEA-000365'
  19. put 'WLSSERVER', 'row3', 'LOG:MSG', 'Server state changed to ADMIN'
  20. put 'WLSSERVER', 'row4', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:19-PM-PDT'
  21. put 'WLSSERVER', 'row4', 'LOG:CATEGORY', 'Notice'
  22. put 'WLSSERVER', 'row4', 'LOG:TYPE', 'WebLogicServer'
  23. put 'WLSSERVER', 'row4', 'LOG:SERVERNAME', 'AdminServer'
  24. put 'WLSSERVER', 'row4', 'LOG:CODE', 'BEA-000365'
  25. put 'WLSSERVER', 'row4', 'LOG:MSG', 'Server state changed to RESUMING'
  26. put 'WLSSERVER', 'row5', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:20-PM-PDT'
  27. put 'WLSSERVER', 'row5', 'LOG:CATEGORY', 'Notice'
  28. put 'WLSSERVER', 'row5', 'LOG:TYPE', 'WebLogicServer'
  29. put 'WLSSERVER', 'row5', 'LOG:SERVERNAME', 'AdminServer'
  30. put 'WLSSERVER', 'row5', 'LOG:CODE', 'BEA-000331'
  31. put 'WLSSERVER', 'row5', 'LOG:MSG', 'Started WebLogic AdminServer'
  32. put 'WLSSERVER', 'row6', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:21-PM-PDT'
  33. put 'WLSSERVER', 'row6', 'LOG:CATEGORY', 'Notice'
  34. put 'WLSSERVER', 'row6', 'LOG:TYPE', 'WebLogicServer'
  35. put 'WLSSERVER', 'row6', 'LOG:SERVERNAME', 'AdminServer'
  36. put 'WLSSERVER', 'row6', 'LOG:CODE', 'BEA-000365'
  37. put 'WLSSERVER', 'row6', 'LOG:MSG', 'Server state changed to RUNNING'
  38. put 'WLSSERVER', 'row7', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:22-PM-PDT'
  39. put 'WLSSERVER', 'row7', 'LOG:CATEGORY', 'Notice'
  40. put 'WLSSERVER', 'row7', 'LOG:TYPE', 'WebLogicServer'
  41. put 'WLSSERVER', 'row7', 'LOG:SERVERNAME', 'AdminServer'
  42. put 'WLSSERVER', 'row7', 'LOG:CODE', 'BEA-000360'
  43. put 'WLSSERVER', 'row7', 'LOG:MSG', 'Server started in RUNNING mode'

 

  • 创建Hive表映射到Hbase表
  1. CREATE EXTERNAL TABLE wlsserver_hbase
  2. ( key string
  3. TIME_STAMP string,
  4. CATEGORY string,
  5. TYPE string,
  6. SERVERNAME string,
  7. CODE string,
  8. MSG string
  9. )
  10. STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
  11. WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,LOG:TIME_STAMP,LOG:CATEGORY,LOG:TYPE,LOG:SERVERNAME,LOG:CODE,LOG:MSG")
  12. TBLPROPERTIES("hbase.table.name" = "WLSSERVER");

 

  • 创建Oracle加载目标表
  1. CREATE TABLE wlsserver
  2. (
  3.     time_stamp VARCHAR2(4000),
  4.     Category VARCHAR2(4000),
  5.     Type VARCHAR2(4000),
  6.     servername VARCHAR2(4000),
  7.     code VARCHAR2(4000),
  8.     msg VARCHAR2(4000)
  9.   );

 

  • 配置Oracle Loader的配置文件
  1. <?xml version="1.0" encoding="UTF-8" ?>
  2. <configuration>
  3.  
  4.    <!-- Input settings -->
  5.    <property>
  6.       <name>mapreduce.inputformat.class</name>
  7.       <value>oracle.hadoop.loader.lib.input.HiveToAvroInputFormat</value>
  8.    </property>
  9.    <property>
  10.       <name>oracle.hadoop.loader.input.hive.databaseName</name>
  11.       <value>default</value>
  12.    </property>
  13.    <property>
  14.       <name>oracle.hadoop.loader.input.hive.tableName</name>
  15.       <value>wlsserver_hbase</value>
  16.    </property>
  17.  
  18.    <!-- Output settings -->
  19.    <property>
  20.       <name>mapreduce.job.outputformat.class</name>
  21.       <value>oracle.hadoop.loader.lib.output.JDBCOutputFormat</value>
  22.    </property>
  23.    <property>
  24.       <name>mapreduce.output.fileoutputformat.outputdir</name>
  25.       <value>oraloadout</value>
  26.    </property>
  27.  
  28.    <!-- Table information -->
  29.    <property>
  30.       <name>oracle.hadoop.loader.loaderMap.targetTable</name>
  31.       <value>WLSSERVER</value>
  32.    </property>
  33.  
  34.    <!-- Connection information -->
  35.    <property>
  36.       <name>oracle.hadoop.loader.connection.url</name>
  37.       <value>jdbc:oracle:thin:@${HOST}:${TCPPORT}:${SID}</value>
  38.    </property>
  39.    <property>
  40.       <name>TCPPORT</name>
  41.       <value>1521</value>
  42.    </property>
  43.    <property>
  44.       <name>HOST</name>
  45.       <value>server1</value>
  46.    </property>
  47.    <property>
  48.       <name>SID</name>
  49.       <value>orcl</value>
  50.    </property>
  51.    <property>
  52.       <name>oracle.hadoop.loader.connection.user</name>
  53.       <value>baron</value>
  54.    </property>
  55.    <property>
  56.       <name>oracle.hadoop.loader.connection.password</name>
  57.       <value>baron</value>
  58.    </property>
  59. </configuration>

 

  • 使用OLH加载 Hbase表到Oracle数据库
  1. hadoop jar $OLH_HOME/jlib/oraloader.jar oracle.hadoop.loader.OraLoader -conf /home/hadoop/OraLoadJobConf-hbase.xml -libjars \
  2. $OLH_HOME/jlib/oraloader.jar,$HIVE_HOME/lib/hive-exec-1.2.1.jar,\
  3. $HIVE_HOME/lib/hive-metastore-1.2.1.jar,$HIVE_HOME/lib/libfb303-0.9.2.jar,\
  4. $HIVE_HOME/lib/hive-hbase-handler-1.2.1.jar,$HBASE_HOME/lib/hbase-client-1.1.2.jar,\
  5. $HBASE_HOME/lib/hbase-server-1.1.2.jar,$HBASE_HOME/lib/hbase-protocol-1.1.2.jar,\
  6. $HBASE_HOME/lib/htrace-core-3.1.0-incubating.jar,$HBASE_HOME/lib/hbase-common-1.1.2.jar,\
  7. $HBASE_HOME/lib/netty-all-4.0.23.Final.jar

 

输出结果如下：

  1. Oracle Loader for Hadoop Release 3.5.0 - Production
  2. Copyright (c) 2011, 2015, Oracle and/or its affiliates. All rights reserved.
  3. 15/12/21 10:37:22 INFO loader.OraLoader: Oracle Loader for Hadoop Release 3.5.0 - Production
  4. Copyright (c) 2011, 2015, Oracle and/or its affiliates. All rights reserved.
  5. 15/12/21 10:37:22 INFO loader.OraLoader: Built-Against: hadoop-2.2.0 hive-0.13.0 avro-1.7.6 jackson-1.8.8
  6. 15/12/21 10:37:50 INFO loader.OraLoader: oracle.hadoop.loader.loadByPartition is disabled because table: WLSSERVER is not partitioned
  7. 15/12/21 10:37:50 INFO loader.OraLoader: oracle.hadoop.loader.enableSorting disabled, no sorting key provided
  8. 15/12/21 10:37:50 INFO loader.OraLoader: Reduce tasks set to 0 because of no partitioning or sorting. Loading will be done in the map phase.
  9. 15/12/21 10:37:50 INFO output.DBOutputFormat: Setting map tasks speculative execution to false for : oracle.hadoop.loader.lib.output.JDBCOutputFormat
  10. 15/12/21 10:37:52 WARN loader.OraLoader: Sampler is disabled because the number of reduce tasks is less than two. Job will continue without sampled information.
  11. 15/12/21 10:37:52 INFO loader.OraLoader: Submitting OraLoader job OraLoader
  12. 15/12/21 10:37:52 INFO client.RMProxy: Connecting to ResourceManager at server1/192.168.56.101:8032
  13. 15/12/21 10:37:57 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
  14. 15/12/21 10:37:57 INFO metastore.ObjectStore: ObjectStore, initialize called
  15. 15/12/21 10:37:57 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
  16. 15/12/21 10:37:57 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
  17. 15/12/21 10:38:00 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
  18. 15/12/21 10:38:03 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
  19. 15/12/21 10:38:03 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
  20. 15/12/21 10:38:03 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
  21. 15/12/21 10:38:03 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
  22. 15/12/21 10:38:03 INFO DataNucleus.Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
  23. 15/12/21 10:38:04 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL
  24. 15/12/21 10:38:04 INFO metastore.ObjectStore: Initialized ObjectStore
  25. 15/12/21 10:38:04 INFO metastore.HiveMetaStore: Added admin role in metastore
  26. 15/12/21 10:38:04 INFO metastore.HiveMetaStore: Added public role in metastore
  27. 15/12/21 10:38:04 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
  28. 15/12/21 10:38:05 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=wlsserver_hbase
  29. 15/12/21 10:38:05 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=get_table : db=default tbl=wlsserver_hbase
  30. 15/12/21 10:38:06 INFO input.HiveToAvroInputFormat: Row filter: null
  31. 15/12/21 10:38:07 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x62a54948 connecting to ZooKeeper ensemble=localhost:2181
  32. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
  33. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:host.name=server1
  34. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_65
  35. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
  36. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.home=/home/hadoop/jdk1.8.0_65/jre
  37. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/hadoop/hadoop-2.6.2/lib/native
  38. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
  39. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
  40. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
  41. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
  42. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:os.version=3.8.13-55.1.6.el7uek.x86_64
  43. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop
  44. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop
  45. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop
  46. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x62a549480x0, quorum=localhost:2181, baseZNode=/hbase
  47. 15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
  48. 15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
  49. 15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x151c51c160b000d, negotiated timeout = 90000
  50. 15/12/21 10:38:07 WARN mapreduce.TableInputFormatBase: You are using an HTable instance that relies on an HBase-managed Connection. This is usually due to directly creating an HTable, which is deprecated. Instead, you should create a Connection object and then request a Table instance from it. If you don't need the Table instance for your own use, you should instead use the TableInputFormatBase.initalizeTable method directly.
  51. 15/12/21 10:38:07 INFO mapreduce.TableInputFormatBase: Creating an additional unmanaged connection because user provided one can't be used for administrative actions. We'll close it when we close out the table.
  52. 15/12/21 10:38:07 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5b1efaaf connecting to ZooKeeper ensemble=localhost:2181
  53. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x5b1efaaf0x0, quorum=localhost:2181, baseZNode=/hbase
  54. 15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
  55. 15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
  56. 15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x151c51c160b000e, negotiated timeout = 90000
  57. 15/12/21 10:38:07 INFO util.RegionSizeCalculator: Calculating region sizes for table "WLSSERVER".
  58. 15/12/21 10:38:09 INFO metastore.HiveMetaStore: 0: Shutting down the object store...
  59. 15/12/21 10:38:09 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=Shutting down the object store...
  60. 15/12/21 10:38:09 INFO metastore.HiveMetaStore: 0: Metastore shutdown complete.
  61. 15/12/21 10:38:09 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=Metastore shutdown complete.
  62. 15/12/21 10:38:09 INFO mapreduce.JobSubmitter: number of splits:1
  63. 15/12/21 10:38:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1450710904519_0002
  64. 15/12/21 10:38:10 INFO impl.YarnClientImpl: Submitted application application_1450710904519_0002
  65. 15/12/21 10:38:10 INFO mapreduce.Job: The url to track the job: http://server1:8088/proxy/application_1450710904519_0002/
  66. 15/12/21 10:38:28 INFO loader.OraLoader: map 0% reduce 0%
  67. 15/12/21 10:39:12 INFO loader.OraLoader: map 100% reduce 0%
  68. 15/12/21 10:39:17 INFO loader.OraLoader: Job complete: OraLoader (job_1450710904519_0002)
  69. 15/12/21 10:39:17 INFO loader.OraLoader: Counters: 30
  70.         File System Counters
  71.                 FILE: Number of bytes read=0
  72.                 FILE: Number of bytes written=125961
  73.                 FILE: Number of read operations=0
  74.                 FILE: Number of large read operations=0
  75.                 FILE: Number of write operations=0
  76.                 HDFS: Number of bytes read=16889
  77.                 HDFS: Number of bytes written=22053
  78.                 HDFS: Number of read operations=5
  79.                 HDFS: Number of large read operations=0
  80.                 HDFS: Number of write operations=4
  81.         Job Counters
  82.                 Launched map tasks=1
  83.                 Data-local map tasks=1
  84.                 Total time spent by all maps in occupied slots (ms)=45057
  85.                 Total time spent by all reduces in occupied slots (ms)=0
  86.                 Total time spent by all map tasks (ms)=45057
  87.                 Total vcore-seconds taken by all map tasks=45057
  88.                 Total megabyte-seconds taken by all map tasks=46138368
  89.         Map-Reduce Framework
  90.                 Map input records=7
  91.                 Map output records=7
  92.                 Input split bytes=16889
  93.                 Spilled Records=0
  94.                 Failed Shuffles=0
  95.                 Merged Map outputs=0
  96.                 GC time elapsed (ms)=400
  97.                 CPU time spent (ms)=8080
  98.                 Physical memory (bytes) snapshot=196808704
  99.                 Virtual memory (bytes) snapshot=2101088256
  100.                 Total committed heap usage (bytes)=123797504
  101.         File Input Format Counters
  102.                 Bytes Read=0
  103.         File Output Format Counters
  104.                 Bytes Written=22053

 

  • 在Oracle中，验证加载结果
  1. SQL >select count(*) from wlsserver;
  2.  
  3.   COUNT(*)
  4. ----------
  5.          7

 

  • 问题总结
  1. java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.exec.Utilities.copyTableJobPropertiesToConf

    workaround：最初选用了hive-1.2.1版本，该版本太高，降级到hive-1.1.1版本

  2. java.lang.NullPointerException

    workaround：OLH 3.4.0 版本 Bug 21392316 ,升级到 OLH 3.5.0版本

  3. java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.hbase.HBaseSplit not found

    workaround：

    for i in *.jar; do jar -tvf "$i" | grep -Hsi HBaseSplit && echo "$i"; done #找到缺少类的包，并将包加到 -libjars参数后面

 

转载于:https://www.cnblogs.com/panwenyu/p/5058590.html

weixin_30329623
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
hadoop hbase hive 常见问题解决
lhy66的专栏
03-24 12万+
Hadoop常见问题，hbase常见问题，hive常见问题。结合网络资料 和自己遇到的整理。
Hutool连接数据库方法
qq_42287535的博客
10-20 3084
Springboot项目打包后，获取jar包外配置文件数据库信息，连接数据库方法。
spark查询报错org.apache.hadoop.hive.ql.exec.Utilities.copyTableJobPropertiesToConf方法不存在NoSuchMethodError
u011476787的博客
09-10 1127
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.exec.Utilities.copyTableJobPropertiesToConf(Lorg/apache/hadoop/hive/ql/plan/TableDesc;Lorg/apache/hadoop/conf/Configuration;)V
OraLoader v3.6 破解版
01-07
OraLoader是一款Oracle数据导入/导出软件，能够从文本文件导入数据，以及导出数据至文本文件
3、通过datax同步oracle相关-oracle到hdfs
05-29
3、通过datax同步oracle相关-oracle到hdfs 网址：https://blog.csdn.net/chenwewi520feng/article/details/130508968 介绍oracle的相关同步，oracle同步到hdfs中示例以及验证
Hadoop原理与技术Hbase的基本操作
05-24
(3)：hbase shell，进入hbase (4)：list，显示所有表 (5)：解决显示所有表遇到的错误（由于集群处于安全模式，该模式下只能进行只读操作，要解除安全状态才能运行） (6)：create 'tempTable', 'f1', 'f2', 'f3'，...
hbase-2.5.8-hadoop3-bin.tar.gz
最新发布
04-22
就像Bigtable利用了Google文件系统（File System）所提供的分布式数据存储一样，HBase在Hadoop之上提供了类似于Bigtable的能力。HBase是Apache的Hadoop项目的子项目。HBase不同于一般的关系数据库，它是一个适合于非...
hadoop的mapreduce把oracle/mysq导入到hbase和hdfs中的程序
02-28
利用hadoop的mapreduce把oracle/mysql中的数据导入到hbase和hdfs中的两个java程序
hadoop面试题：HBase与BigTable的比较.pdf
06-02
hadoop面试题，HBase与BigTable的比较这是一个很经典的面试题，面试Hadoop必问的问题，值得关注学习。
OraLoader--用sqlldr的Oracle库表数据装载工具
01-08
用过PLSQL developer么？配合起来不错，不用写控制文件了
hive中删除表的错误Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException
热门推荐
风行天下
09-09 3万+
hive中删除表的错误Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:javax.jdo.JDODataStoreException: Iteration request failed :
关于return code X from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
Burgess_Lee的博客
08-28 720
不知道是不是hive-0.12版增强了local mode的原因，在之前版本运行好好的Hive-QL在这个版本上错误频频，折磨一天多以后终于定位到原因，把在内部的总结在这再记录下，希望对遇到同样问题的筒子们有所帮助。部分一 关于return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask Hive升级到0.12版之后，若干...
Hive安装并使用MySQL存储元数据
gakki_smile的博客
08-16 992
Hive默认使用derby存储元数据 derby存在什么缺陷 1.derby不能多个客户端登录 2.derby登录必须在相同目录下，否则可能会找不到所创建的表。 比如在/hive目录下启动hive程序，那么所创建的表就会存储在/hive下面保存。如果在/home下面，所创建的表就会在/home下面保存。 一下操作使用Hadoop用户 下载解压到hive的目录 3. 替换jar包，与hb
Hive配置项的含义详解 hive的配置
多哥仁慈的骆驼
12-20 422
Hive配置项的含义详解 hive的配置： hive.ddl.output.format：hive的ddl语句的输出格式，默认是text，纯文本，还有json格式，这个是0.90以后才出的新配置； hive.exec.script.wrapper：hive调用脚本时的包装器，默认是null，如果设置为python的话，那么在做脚本调用操作时语句会变为python ，null的话就是直接执行；
启动hive时报错： org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark session
weixin_55842692的博客
12-12 2534
h启动hive时报错： Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session a2d32dbe-d48c-486f-be8b-2f5c75ffe182 日志监控： java.util.concurrent.ExecutionException: java.util.concurren.
Hive与HBase联合操作异常
haison_first的专栏
11-21 1080
环境配置： 1.hadoop2.2.0 2.hive 0.12 3.hbase0.94.5(基于hadoop2.2.0重新编译)
Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
微电子学与固体电子学-俞驰
07-22 3946
这个问题先检查mysql中的hive数据库是否有一大堆的表格， 如果没有的话，需要先 一. hive元数据库初始化，执行 schematool -dbType mysql -initSchema （schematool是hive文件夹下面的一个文件） 这个初始化后的结果在mysql种的hive数据库中 二． 然后检查hive-site.xml中是否数据库的用户名和密码弄错了  ...
在信贷业务中同时用到Oracle和HADOOP怎么使用
06-09
3. 使用Hive作为Hadoop的数据仓库，将Oracle中的数据导入Hive中进行进一步的处理和分析。 4. 使用Sqoop工具将数据从Oracle中导入到Hadoop中，或者将处理后的数据从Hadoop中导出到Oracle中。 5. 使用Spark作为...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值