HADOOP-ORACLE: 装载Hadoop 文件到数据库

问题:

1.在读取配置文件时，无法找到mapreduce.outputformat.class 的对应值

12/12/12 17:30:11 ERROR loader.OraLoader: mapreduce.outputformat.class is invalid

oracle.hadoop.loader.OraLoaderException: missing or invalid property "mapreduce.outputformat.class" value

解决：在执行hadoop   ${OLH_JAR}    oracle.hadoop.loader.OraLoader -conf /home/oracle/OLH/MyConf.xml 时，将MyConf.xml文件的路径写成了 HDFS文件系统的路径，将其改为LINUX 下的路径。

 

2.在进行连接时，无法连接监听器

解决:  没有配置好，本机的lisener,tnsname ,sqlnet 文件

 

3.在进行Loader时需要的文件， IN_DIRECTORY,OUT_DIRECTORY, DATAFILE,

重要的对象:

HDFS:

start:

mapred.input.dir,

   mapreduce.outputformat.class

   mapreduce.inputformat.class

  mapreduce.output.dir

end:

 oracle.hadoop.loader.loaderMapFile

oracle.hadoop.loader.targetTable

oracle.hadoop.loader.input.fieldNames

oracle.hadoop.loader.connection.url

 

-    < name > mapreduce.inputformat.class </ name >

  < value > oracle.hadoop.loader.lib.input.DelimitedTextInputFormat </ value >

 

  < name > mapred.input.dir </ name >

  < value > /user/root/olh_lab_in/data.dat </ value >

 

-    < name > mapreduce.outputformat.class </ name >

  < value > oracle.hadoop.loader.lib.output.JDBCOutputFormat </ value >

 

< name > mapred.output.dir </ name >

  < value > /user/root/olh_lab_out </ value >

 

- 

  < name > oracle.hadoop.loader.loaderMapFile </ name >

  < value > file:///home/oracle/OLH/loaderMap_exercise1.xml </ value >

    

<name>oracle.hadoop.loader.targetTable</name>
<value>SCOTT.OLH_TABLE</value>

 

<name>oracle.hadoop.loader.input.fieldNames</name>
<value>col1,col2,col3,col4</value>
  

 

< name > oracle.hadoop.loader.connection.url </ name >

  < value > jdbc:oracle:thin:@${HOST}:${TCPPORT}/${SERVICE_NAME} </ value >

 

< name > TCPPORT </ name >

  < value > 1521 </ value >

 

< name > HOST </ name >

  < value > hadoop.oracle </ value >

 

< name > ORACLE_SID </ name >

  < value > hadoop </ value >

 

< name > SERVICE_NAME </ name >

  < value > hadoop </ value >

 

< name > oracle.hadoop.loader.connection.user </ name >

  < value > SCOTT </ value >

  < description > AvroSchemaAwareDBInputFormat wants this case sensitive! (cause mapred.jdbc.username points here) </ description >

 

< name > oracle.hadoop.loader.connection.password </ name >

  < value > tiger </ value >

 

 

http://docs.oracle.com/cd/E27101_01/doc.10/e27365/olh.htm

 

 

[oracle@hadoop admin]$ hadoop dfs -rmr /user/root/olh_lab_out
Deleted hdfs://hdnode1m:9000/user/root/olh_lab_out
[oracle@hadoop admin]$ hadoop jar $OLH_HOME/jlib/oraloader.jar oracle.hadoop.loader.OraLoader -conf /home/oracle/OLH/MyConf.xml
Oracle Loader for Hadoop Release 2.0.0 - Production

 

Copyright (c) 2011, 2012, Oracle and/or its affiliates. All rights reserved.

 

13/03/07 19:38:38 INFO loader.OraLoader: Oracle Loader for Hadoop Release 2.0.0 - Production

 

Copyright (c) 2011, 2012, Oracle and/or its affiliates. All rights reserved.

 

13/03/07 19:38:38 INFO loader.OraLoader: Built-Against: hadoop-0.20.2 hive-0.7.1-cdh3u3 avro-1.6.3 jackson-1.8.8
13/03/07 19:38:39 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
13/03/07 19:38:40 INFO loader.OraLoader: oracle.hadoop.loader.loadByPartition is disabled because table: OLH_TABLE is not partitioned
13/03/07 19:38:40 INFO loader.OraLoader: oracle.hadoop.loader.enableSorting disabled, no sorting key provided
13/03/07 19:38:40 INFO output.DBOutputFormat: Setting reduce tasks speculative execution to false for : oracle.hadoop.loader.lib.output.JDBCOutputFormat
13/03/07 19:38:40 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
13/03/07 19:38:40 WARN loader.OraLoader: Sampler error: the number of reduce tasks must be greater than one; the configured value is 1 . Job will continue without sampled information.
13/03/07 19:38:40 INFO loader.OraLoader: Sampling time=0D:0h:0m:0s:13ms (13 ms)
13/03/07 19:38:40 INFO loader.OraLoader: Submitting OraLoader job OraLoader
13/03/07 19:38:42 INFO input.FileInputFormat: Total input paths to process : 1
13/03/07 19:38:42 INFO input.FileInputFormat: Total input paths to process : 1
13/03/07 19:38:42 INFO mapred.MapTask: io.sort.mb = 100
13/03/07 19:38:43 INFO mapred.MapTask: data buffer = 79691776/99614720
13/03/07 19:38:43 INFO mapred.MapTask: record buffer = 262144/327680
13/03/07 19:38:43 INFO mapred.MapTask: Starting flush of map output
13/03/07 19:38:43 INFO mapred.MapTask: Finished spill 0
13/03/07 19:38:43 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
13/03/07 19:38:43 INFO mapred.LocalJobRunner:
13/03/07 19:38:43 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
13/03/07 19:38:43 INFO mapred.LocalJobRunner:
13/03/07 19:38:43 INFO mapred.Merger: Merging 1 sorted segments
13/03/07 19:38:43 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1196 bytes
13/03/07 19:38:43 INFO mapred.LocalJobRunner:
13/03/07 19:38:43 INFO output.DBOutputFormat: conf prop: defaultExecuteBatch: 100
13/03/07 19:38:43 INFO output.DBOutputFormat: conf prop: loadByPartition: false
13/03/07 19:38:43 INFO output.DBOutputFormat: Insert statement: INSERT INTO "SCOTT"."OLH_TABLE" ("COL1", "COL2", "COL3", "COL4") VALUES (?, ?, ?, ?)
13/03/07 19:38:43 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
13/03/07 19:38:43 INFO mapred.LocalJobRunner:
13/03/07 19:38:43 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
13/03/07 19:38:43 INFO output.JDBCOutputFormat: Committed work for task attempt attempt_local_0001_r_000000_0
13/03/07 19:38:43 INFO output.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to /user/root/olh_lab_out
13/03/07 19:38:43 INFO mapred.LocalJobRunner: reduce > reduce
13/03/07 19:38:43 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
13/03/07 19:38:43 INFO loader.OraLoader: map 100% reduce 100%
13/03/07 19:38:43 INFO loader.OraLoader: Job complete: OraLoader (null)
13/03/07 19:38:43 INFO loader.OraLoader: Counters: 14
        FileSystemCounters
                FILE_BYTES_READ=18885058
                FILE_BYTES_WRITTEN=55486
                HDFS_BYTES_READ=82429
                HDFS_BYTES_WRITTEN=18957078
        Map-Reduce Framework
                Combine input records=0
                Combine output records=0
                Map input records=20
                Map output bytes=1154
                Map output records=20
                Reduce input groups=1
                Reduce input records=20
                Reduce output records=20
                Reduce shuffle bytes=0
                Spilled Records=40