2013-03-01

1 执行jar文件
java.util.zip.ZipException: error in opening zip file
文件看看存在不，不存在的话就看看原来和现在所在位置的jdk版本一致不
2 50030页面打不开，50070可以打开，hadoop上传下载文件都没有问题，就是不能执行作业，会报9001连接不上，被拒绝的错误。各个节点的jps显示进程数也是正常的，但主节点查看mapred对应的日志，发现有报错，一个文件找不到。原来是有个mapred.hosts.exclude文件的名字写错了，改好配置文件后，重启hadoop集群，一切正常了。
3 hive建外部表时，location不能有2级目录，直接指向到有日志文件存在的那一层
4 使用insert into select from 语句报错：
2013-03-01 05:35:31.378 GMT Thread[main,5,main] java.io.FileNotFoundException: derby.log (Permission denied)
2013-03-01 05:35:31.460 GMT Thread[main,5,main] Cleanup action starting
ERROR XBM0H: Directory /root/TempStatsStore cannot be created.
        at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
        at org.apache.derby.impl.services.monitor.StorageFactoryService$9.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown Source)
        at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
        at org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(Unknown Source)
        at org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.createDatabase(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(Unknown Source)
        at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source)
        at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
        at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)
        at java.sql.DriverManager.getConnection(DriverManager.java:582)
        at java.sql.DriverManager.getConnection(DriverManager.java:207)
        at org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.init(JDBCStatsPublisher.java:259)
        at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:445)
        at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Cleanup action completed
2013-03-01 05:35:31.466 GMT Thread[main,5,main] Cleanup action starting
java.sql.SQLException: Failed to create database 'TempStatsStore', see the next exception for details.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.createDatabase(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(Unknown Source)
        at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source)
        at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
        at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)
        at java.sql.DriverManager.getConnection(DriverManager.java:582)
        at java.sql.DriverManager.getConnection(DriverManager.java:207)
        at org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.init(JDBCStatsPublisher.java:259)
        at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:445)
        at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.sql.SQLException: Failed to create database 'TempStatsStore', see the next exception for details.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
        ... 30 more
Caused by: java.sql.SQLException: Directory /root/TempStatsStore cannot be created.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
        ... 27 more
Caused by: ERROR XBM0H: Directory /root/TempStatsStore cannot be created.
        at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
        at org.apache.derby.impl.services.monitor.StorageFactoryService$9.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown Source)
        at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
        at org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(Unknown Source)
        at org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Unknown Source)
        ... 27 more
======解决：
修改hive-site.xml文件中以下属性：

<property>
  <name>hive.stats.dbclass</name>
  <!--value>jdbc:derby</value-->
  <value>jdbc:mysql</value>
  <description>The default database that stores temporary hive statistics.</description>
</property>


<property>
  <name>hive.stats.autogather</name>
  <value>true</value>
  <description>A flag to gather statistics automatically during the INSERT OVERWRITE command.</description>
</property>


<property>
  <name>hive.stats.jdbcdriver</name>
 <!-- <value>org.apache.derby.jdbc.EmbeddedDriver</value>-->
  <value>com.jdbc.mysql.Driver</value>
  <description>The JDBC driver for the database that stores temporary hive statistics.</description>
</property>


<property>
  <name>hive.stats.dbconnectionstring</name>
  <!--<value>jdbc:derby:;databaseName=TempStatsStore;create=true</value>-->
  <value>jdbc:mysql://mymaster:3306/TempStatsStore</value>
</property>
然后退出hive控制台，再重新进入。
5 将产生的结果暂存在一个表中，之后便可以用sqoop导出到mysql中。 
insert into table pvresult select requrl,ip,count(*) from test group by requrl,ip;
在创建pvresult表时，如果不指定分隔符，默认会按照ctrl-a分隔，查出来的数据再保存进这个新表
6 sqoop执行导出时报错：当以某用户身份执行sqoop命令时，要在该用户有权限的目录作为当前目录，执行sqoop命令才能成功，因为会涉及到往当前目录下写文件。
java.util.NoSuchElementException 
因为没指定数据字段的分隔符。sqoop解析文件的字段与MySql数据库的表的字段对应不上造成的。因此需要在执行的时候给sqoop增加参数，告诉sqoop文件的分隔符，使它能够正确的解析文件字段。 --input-fields-terminated-by '\t'  。主要看你hive表中的数据是怎么分割的，默认是\0001(ctrl-a)