1、编写mapreduce无法直接把结果写入oracle
1）继承DBOuputFormat类，把constructQuery方法重写，其中代码不变，把里面的query.append(");")改为query.append(")");
2）重新定义一个Format类，把DBOutFormat中的代码拷贝过去，然后修改constructQuery方法，把里面的(;)改为()，跟上面一样
2、Hadoop内置InputFormat支持的文件压缩格式情况
| 压缩格式 | split | native | 压缩率 | 速度 | 是否hadoop自带 | linux命令 | 换成压缩格式后，原来的应用程序是否要修改 |
|---|---|---|---|---|---|---|---|
| gzip | 否 | 是 | 很高 | 比较快 | 是，直接使用 | 有 | 和文本处理一样，不需要修改 |
| lzo | 是 | 是 | 比较高 | 很快 | 否，需要安装 | 有 | 需要建索引，还需要指定输入格式 |
| snappy | 否 | 是 | 比较高 | 很快 | 否，需要安装 | 没有 | 和文本处理一样，不需要修改 |
| bzip2 | 是 | 否 | 最高 | 慢 | 是，直接使用 | 有 | 和文本处理一样，不需要修改 |
3、hive-e执行出现了warn
执行hive -S -e 'select count(1) from ori.t_cust_req_ori'，出现了
15/03/0610:00:10 WARN conf.HiveConf: DEPRECATED: Configuration propertyhive.metastore.local no longer has any effect. Make sure to provide a validvalue for hive.metastore.uris if you are connecting to a remote metastore.
在0.10 0.11或者之后的HIVE版本hive.metastore.local 属性不再使用。
网上给出的解决方案： <property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>
删除掉。
再次登录 警告就消失了。
4、hive本地模式执行失败（未解决）
set hive.exec.mode.local.auto = true;
然后执行一个数据量比较小的查询，让hive使用本地模式查询，然后就报错了，错误信息如下:
2015-03-18 16:52:40,444 INFO [main]: mapreduce.JobSubmitter(JobSubmitter.java:submitJobInternal(537)) - Cleaning up the staging areafile:/tmp/hadoop-root/mapred/staging/root1019557723/.staging/job_local1019557723_0001
2015-03-18 16:52:40,448 WARN [main]: security.UserGroupInformation(UserGroupInformation.java:doAs(1645)) - PriviledgedActionException as:root(auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist:hdfs://hadoop01:8020/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/hive-hbase-handler-0.13.1-cdh5.3.1.jar
2015-03-18 16:52:40,448 WARN [main]: security.UserGroupInformation(UserGroupInformation.java:doAs(1645)) - PriviledgedActionException as:root(auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: hdfs://hadoop01:8020/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/hive-hbase-handler-0.13.1-cdh5.3.1.jar
2015-03-18 16:52:40,450 ERROR [main]:mr.ExecDriver (SessionState.java:printError(565)) - Job Submission failed withexception 'java.io.FileNotFoundException(File does not exist:hdfs://hadoop01:8020/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/hive-hbase-handler-0.13.1-cdh5.3.1.jar)'
java.io.FileNotFoundException: File doesnot exist: hdfs://hadoop01:8020/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/hive-hbase-handler-0.13.1-cdh5.3.1.jar
atorg.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093)
atorg.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
atorg.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
atorg.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
atorg.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
atorg.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
atorg.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
atorg.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
atorg.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:267)
atorg.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:388)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:481)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:564)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:559)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
atorg.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:559)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:550)
atorg.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420)
atorg.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:740)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
5、map方法中，注意context.write
context.write($1,$2)，$1与$2都要是对象，记得其中$2不能为NULL
6、oozie不能执行主机上的shell
这边集群hadoop发行版是cdh5，我使用hue来设计一个简单的基于oozie的job（就是执行主机上的ls命令），oozie服务是在hadoop01机器，job执行hadoop01上的命令，并且整个集群是无密码登陆的，那么问题来了，执行的时候报了以下错误：
org.apache.oozie.action.ActionExecutorException:AUTH_FAILED: Not able to perform operation [ssh -o PasswordAuthentication=no -oKbdInteractiveDevices=no -o StrictHostKeyChecking=no -o ConnectTimeout=20@root@hadoop01 mkdir -poozie-oozi/0000003-150310113935079-oozie-oozi-W/Ssh--ssh/ ] | ErrorStream:Warning: Permanently added hadoop01 (RSA) to the list of known hosts.
Permission denied(publickey,gssapi-with-mic,password).
atorg.apache.oozie.action.ssh.SshActionExecutor.execute(SshActionExecutor.java:590)
atorg.apache.oozie.action.ssh.SshActionExecutor.start(SshActionExecutor.java:204)
atorg.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
atorg.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
atorg.apache.oozie.command.XCommand.call(XCommand.java:281)
atorg.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
atorg.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
atorg.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
atjava.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Not able toperform operation [ssh -o PasswordAuthentication=no -o KbdInteractiveDevices=no-o StrictHostKeyChecking=no -o ConnectTimeout=20 @root@180.200.2.36 mkdir -poozie-oozi/0000003-150310113935079-oozie-oozi-W/Ssh--ssh/ ] | ErrorStream:Warning: Permanently added '180.200.2.36' (RSA) to the list of known hosts.
Permission denied (publickey,gssapi-with-mic,password).
atorg.apache.oozie.action.ssh.SshActionExecutor.executeCommand(SshActionExecutor.java:340)
atorg.apache.oozie.action.ssh.SshActionExecutor.setupRemote(SshActionExecutor.java:373)
atorg.apache.oozie.action.ssh.SshActionExecutor$1.call(SshActionExecutor.java:206)
atorg.apache.oozie.action.ssh.SshActionExecutor$1.call(SshActionExecutor.java:204)
atorg.apache.oozie.action.ssh.SshActionExecutor.execute(SshActionExecutor.java:548)
...10 more
解决：在cdh中，打开oozie的配置项，在高级那边，设置为root，简单粗暴
7、Oozie安装问题
另外，需要访问oozie web ui的还需以下步骤：
访问webapp console的情况：http://hadoop01:11000/oozie/，说没有安装共享库，那只能下载ext-2.2.zip，并且把它解压到/var/lib/oozie的目录当中，要不然/var/lib/oozie/tomcat-deployment/webapps/oozie下的软连接ext-2.2-> /var/lib/oozie/ext-2.2找不到ext目录的
194
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
